代做COMP5318/COMP4318 Machine Learning and Data Mining Semester 1, 2025 Assignment 2: Image Classifica

COMP5318/COMP4318 Machine Learning and Data Mining

Semester 1, 2025

Assignment 2: Image Classification

Key information

Deadlines

Submission: Monday week 11 (12 May), 11.59pm

Late submissions policy

Late submissions are allowed for up to 3 days late. A penalty of 5% per day late will apply. Assignments more than 3 days late will not be accepted (i.e. will get 0 marks). The day cut-off time is 11:59pm.

Marking

This assignment is worth 25 marks = 25% of your final mark. It consists of two components: code (10 marks) and report (15 marks). A marking guide for both the code and report is included at the end of this document.

The assignment can be completed in groups of 2 or 3 students. No more than 3 students are allowed. See the submission details section for more information about how to submit.

Submission

Three files are required to be submitted in the relevant submission portals on Canvas:

-      Your report as a .pdf file

-      Your Jupyter notebook as a .ipynb file

-      Your Jupyter notebook as a .pdf file

A pdf of your Jupyter notebook can be generated using File>Download as>PDF or Print Preview > Save as PDF or File->Download as >HTML, then open the html file and save it as pdf.

Name your files with the following format:

-      Report:

o A2-report-SID1-SID2-SID3.pdf

-      Code:

o a2-code-SID1-SID2-SID3.ipynb

o a2-code-SID1-SID2-SID3.pdf

where SID1, SID2 and SID3 are the SIDs of the three students in your group.

Please do not include your names anywhere in your submissions or the file names since the marking is anonymous.

Before you submit, you need to create a group in Canvas. Under the “People” page on Canvas, select the “A2 groups” tab. You and your group partners should choose one of the empty groups listed under this tab, and join it. Groups have a maximum of 3 members.  The assignment should be completed in groups of 2 or 3 students.

Important: The "A1 group-set1" and "A1 group-set2"  registration  is for Assignment  1, the  "A2 groups" registration is for Assignment 2. You need to register a group under "A2 groups" even if you work with the same partner as in Assignment 1. Please be careful as otherwise your mark may not be recorded correctly. There may also be a mark deduction for not following the instructions.

Then you need to submit your assignment on behalf of the group in the corresponding submission box. Only one student from the group needs to submit, not all.

Code information

The code for this assignment should be written in Python in the Jupyter Notebook environment. Please follow  the  structure  in  the  template  notebook  provided  as  part  of  the  assignment  materials.  Your implementation  of  the  algorithms  should   predominantly  utilise  the  same  suite  of   libraries  we   have introduced in the tutorials (Keras, scikit-learn, numpy, pandas etc.). Other libraries may be utilised for minor functionality such as  plotting,  however  please specify any dependencies at the  beginning of your code submission. While most of your explanation and justification can be included in the report, please ensure your code is well formatted, and that there are sufficient comments or text included in the notebook to explain the cells.

You may choose to run your code locally or on a cloud service (such as Google Colaboratory), however your final submission needs to be able to run on a local machine. Please submit your notebook with the cell output  preserved  and  ensure  that  all  the  results  presented  in  your  report  are  demonstrated  in  your submitted notebook. Your code may also be rerun by your marker, so please ensure there are no errors in your submitted code and it can be run in order.

Task

In this  assignment,  you will  implement  and  compare  several  machine  learning  algorithms,  including  a Multilayer Perceptron (MLP) and Convolutional Neural Network (CNN), on an image classification task. You will need to demonstrate your understanding of a full machine learning pipeline, including data exploration and  pre-processing,  model  design,  hyperparameter  tuning,  and  interpreting  results.  Moreover,  the assignment will require you to consolidate your knowledge from the course so far to effectively discuss the important differences between the algorithms.

While better performance is desirable, it is not the main objective of this assignment. Since chasing the best possible  performance  may  require  large  models  and  specialised  GPU  hardware,  you  should  focus  on thoroughly justifying your decisions and analysing your results. Please see the marking criteria at the end of the specification for how you will be assessed. There are no marks allocated for reaching a particular performance or having highly complex models.

1.      Code

As mentioned above, your code submission for the assignment should be an .ipynb Jupyter notebook in similar fashion to the tutorials, including well commented code and sufficient text to explain the cells. Please follow the structure provided in the template notebook available as part of the assignment materials.

Data loading, pre-processing, and exploration

The dataset you will use in this assignment is PathMNIST. It contains 28x28 colour images of microscope slides of normal and abnormal body tissues. More information on the dataset and the original source can be found in the associated paper here: https://www.nature.com/articles/s41597-022-01721-8 . The dataset is licensed under CC BY 4.0 (attribution is provided below). Note that we have provided a subset of this dataset for the assignment, with different splits to the original.

Please refer to the provided dataset on Canvas rather than downloading from the MedMNIST site. Code to load these files is provided in the template notebook.

The images in this dataset have relatively low dimensionality to aid keeping your runtimes  short. You can increase/decrease the dimensionality of the data or make use of a subset of the training data as required, with justification in the report.

To better understand the properties of the data and the preprocessing that may be appropriate, you should begin with some exploration of the data. You may like to explore what the different classes are, the number of examples from each class, and consider characteristics of the images such as whether they are centred, size of different features in the images, pixel intensities across different images, key differences between classes etc. Explore if there are factors that might make the task more difficult, such as classes with similar features. You should include anything you feel is relevant in this section.

Based on your insight from the data exploration and/or with reference to other sources, you should apply   appropriate preprocessing techniques. Your choice of preprocessing needs to be justified in the report. You may apply different preprocessing techniques for the different algorithms, with justification in the report.

Consider if you need to make any additional splits of the data and think carefully about how each part of the data should be utilised to evaluate hyperparameter combinations and compare the performance of the different models.

Algorithm design and setup

You will be required to design and implement three algorithms that we have covered in the course using the sklearn and/or keras libraries, in order to investigate their strengths and weaknesses.

1.    An appropriate algorithm of your choice from the first 6 weeks of the course

2.    A feedforward multilayer perceptron neural network (MLP)

3.    A convolutional neural network (CNN)

In this  section,  implement  an  instance  of  each  model  before  tuning  hyperparameters,  and  set  up  any functions you may require to tune hyperparameters in the next section.

Due to runtime constraints, it is not feasible to consider every possible neural network  architecture when designing your models, however you should justify your design decisions in the report, and you may wish to conduct a few rough experiments to converge on a reasonable design. Remembering that the focus of the assignment is to demonstrate your understanding of and compare the algorithms, not to achieve state-of- the-art   performance,   keep   your    models   appropriately   small   such   that   you   can   complete   the hyperparameter tuning in a reasonable time on your hardware of choice.

Although you may like to reference external sources when designing your algorithms, you must implement your  neural  network  models  yourself,  rather  than  import  prebuilt  models  from  Keras  (such  as  those available in keras.applications).

Hyperparameter tuning

Perform. a search over relevant hyperparameters for each algorithm using an appropriate search strategy of your choice (eg. cross validation, validation set grid search, Bayesian search, etc.). You may use scikit-learn and/or keras-tuner to perform. your searches, and you are suggested to look through the documentation of these packages to see the available utilities and decide on your implementation. Remember there are many factors that can affect the runtime of your search, such as the  number of epochs, optimisers, and the particular hyperparameters included in the search. Please ensure you search over the following:

o Algorithm of your choice from the first 6 weeks:

o You will need to choose an appropriate set of hyperparameters to search over based on what was covered in the tutorials or other references

o Multilayer feedforward and convolutional neural networks:

o Tune over at least 3 different hyperparameters

You will need to justify your search method, choices of hyperparameters to search over, and the values included in the search as part of your report (see below).

Keep a record of the runtimes and results with each hyperparameter combination (you may need to consult documentation   to   see   how   to   extract   this   information)   and   use   these   to    produce   appropriate visualisations/tables of the trends in your hyperparameter search to aid the discussion in your report.

Please preserve the output of these cells in your submission and keep these hyperparameter search cells independent from the other cells of your notebook to avoid needing to rerun them, as markers will not be able to run this code (it will take too long), i.e. ensure the later cells can be run if these grid search cells are skipped.

Final models

After selecting the best set of hyperparameters for each model, include cells which train the models with the selected hyperparameters independently of the parameter search cell, and use these implementations to compare the performance (and other relevant properties) of the different models using the test set.

2.      Report

An approximate outline of the  report  requirements  is  provided  below,  but  make  sure  to  reference the marking criteria also for the mark distribution. You may write your report in MS Word or LaTeX, but please ensure that it is well-formatted and submit it as a pdf.

Please stick to the structure (headings and subheadings) outlined below, as these align with the marking criteria.

Please do not include screenshots of your code in the report. The report should not be focussed on your code and describing exactly how you have implemented things, but rather on the content outlined below.

Introduction

State the aim of your study and outline the importance of your study. You can consider both the importance of this dataset itself, but also the importance of comparing algorithms and their suitability for the task more generally.

Data

In this section, you should describe the dataset and pre-processing.

Data description and exploration. Describe the data, including all its important characteristics, such as the number  of  samples,  classes,   dimensions,   and  the  original  source  of  the   images.   Discuss  your   data exploration, including characteristics/difficulties as described in the relevant section above and anything you consider relevant. Where relevant, you may wish to include some sample images to aid this discussion. For information about the original dataset, such as the class names, the MedMNIST site (https://medmnist.com) and papers included in the dataset attribution below may be useful.

Pre-processing. Justify your choice of pre-processing either through your insights from the data exploration or with reference to other sources. Explain how the preprocessing techniques work, their effect/purpose, and any choices in their application. If you have not performed pre-processing or have intentionally omitted possible preprocessing techniques after consideration, justify these decisions.

Methods

In this section, you should explain the classification methods you used.

Theory.  For  each  algorithm,  explain  the  main  theoretical  ideas  (this  will  be  useful  as  a  framework  for comparing them in the rest of the report). Explain why you chose your particular algorithm from the first six weeks.

Strengths and weaknesses. Describe the relative strengths and weaknesses of the algorithms from a theory perspective.   Consider   factors   such   as   performance,   overfitting,    runtime,   number   of   params   and interpretability. Explain the reasons; e.g. don’t simply state that CNNs perform better on images, but explain why this is the case.

Architecture and hyperparameters – State and explain the chosen architectures or other relevant design choices you made in your implementation (e.g. this is the place to discuss your particular neural network configurations). Describe the hyperparameters you will tune over, the values included in the search, and outline your search method. Briefly explain what each hyperparameter controls and the expected effect on the algorithm. For example, consider the effects of changing the learning rate, or changing the stride of a convolutional   layer.   Justify   why   you    have   made   these    choices   about   the    search   method,    and hyperparameters included.

Results and discussion

In this section, you should present and discuss your results. Please do not include screenshots of raw code outputs when presenting your results. Instead tabulate/plot any results in a manner more appropriate for presentation in the report.

Begin with the hyperparameter tuning results. Include appropriate tables or graphs to illustrate the trends (performance, runtime etc.) across different hyperparameter values. Discuss the trends and provide possible explanations for their observation. Consider if the results aligned with your predictions.

Next,   present   a   comparison   of   the   models   you   have   implemented   for   this  task   (with   the   best hyperparameters found). Include a table showing the best hyperparameter combination for each algorithm, the  performance on the test set  (e.g. accuracy and other  performance  measures),  and the  runtime(s). Analyse and discuss the results, referring back to the theoretical properties and strengths/weaknesses of the classifiers discussed in the Methods section. Consider if the results aligned with your expectations. What factors influenced the runtime (time per epoch, total number of epochs required etc.)?

Include anything you consider interesting and/or relevant. For example, you may like to comment on the types of mistakes particular models made via confusion matrices etc.

Conclusion

Summarise  your  main  findings,  mention  any  limitations,  and  suggest  future  work.  When  making  your conclusions, consider not only accuracy, but also factors such as runtime and interpretability. Ensure your future work suggestions are concrete (e.g. not in the spirit of “try more algorithms”) and justify why they would be appropriate.

Reflection

Write one or two paragraphs describing the most important thing that you have learned while completing the assignment. Every student should write their own reflection.

References

Include references to any sources you have utilised in completing the code and/or report. You may choose any appropriate academic referencing style, such as IEEE.