联系我们： 手动添加方式: 微信>添加朋友>企业微信联系人>13262280223 或者 QQ: 1483266981

This paper consists of EIGHT questions. Answer ALL EIGHT questions.
Marks are distributed as follows:

?Q1: 25 marks

?Q2: 5 marks

?Q3: 25 marks

?Q4: 6 marks

?Q5: 5 marks

?Q6: 5 marks

?Q7: 5 marks

?Q8: 24 marks

Include any relevant working and/or reasoning for your answers—unjustified answers will receive no marks even if they are numerically correct. You may use software such as NumPy for linear algebraic calculations, getting PDF values etc, but you must explain what you are doing without code. Keep answers concise and to the point.

Marks for each part of each question are indicated in square brackets. Calculators are permitted.

x1 1 2 3 1 2 3 1 2 3
x2 1 1 1 2 2 2 3 3 3
y 10.28 8.87 5.73 12.52 11.31 9.58 14.43 13.17 11.45
Table 1: Training data for Question 1

1.Table 1 shows sample data drawn from a simple linear model with 2 feature dimensions and additive Gaussian noise:
y = w1x1 + w2x2 + b + ε, ε ～ N (0, σ2)
a.Convert this into a problem with no intercept by centring the data.

i.Construct a design matrix X of the centred features.
[2 marks]
x1 and x2 both have a mean of 2, so the design matrix (transposed to save space) becomes:

ii.Estimate b.

XT = ?1 0 1 ?1 0 1 ?1 0 1
?1 ?1 ?1 0 0 0 1 1 1

[2 marks]

Since the expected contribution of the centred features is zero, all the offset in y must come now from the intercept. So

b = yˉ ≈ 10.82

iii.Centre y.

This just entails subtracting b, so:

[1 marks]

yT = ?0.54 ?1.95 ?5.09 1.7 0.49 ?1.24 3.61 2.35 0.63

[Question 1 cont. on next page]

[Question 1 cont.]
b.Estimate the feature weights w1 and w2 by ordinary least squares regression.
i.Write down a pair of linear equations in [w1, w2] that define the least squares fit.

We get these from the OLS normal equation, XTXw = XTy.
XTX = 6 0 XTy = ?10.47

[4 marks]

So our pair of equations are:

0 6

6w1 = ?10.47
6w2 = 14.17

14.17

ii.Solve the equations for w1 and w2.
This is trivially solved since each equation has only one variable:
w1 = ?1.74
w2 = 2.36

iii.What does the form of the equations tell you about the features?

[1 marks]

[3 marks]

For centred data, XTX = nΣ, ie a multiple of the covariance matrix. Since it is diag- onal, the features are uncorrelated. (Orthogonal or linearly-independent will also do here, though not strictly equivalent.)

iv.Estimate the amount of additive noise, σ.

[4 marks]

Predict y? using fitted weights, use that to calculate the residuals and then estimate σ
from those. (We ignore b here, students might include, which is fine.)
y?T = 0.62 2.36 4.11 1.74 0 1.74 4.11 2.36 0.62
(y ? y?)T = ?0.08 ?0.42 0.98 0.04 ?0.49 ?0.51 0.49 0.01 ?0.02
σ = std(y ? y?) = 0.46
v.Would you expect the model parameters estimated by this method (w1, w2, b, σ) to have the same values as you would have obtained from fitting a model with an intercept to the original uncentred data?
[2 marks]
The values for w1, w2 and σ should be consistent for both approaches. The value for b will be different: with uncentred data, the offsets in x1 and x2 will contribute to the offset in y.

[Question 1 cont. over page]

[Question 1 cont.]

c.Re-estimate the weights w1 and w2 by ridge regression with λ = 1.
i.Adapt your equations from part b to include the ridge penalty and solve for the regularised weights.
[4 marks]
The ridge penalty modifies the OLS equations as
(XTX + λI)w = XTy
ie we just need to add λ = 1 to the diagonal coefficients. Since XTX is already diagonal, this produces a simple change:
7w1 = ?10.47
7w2 = 14.17

Hence:

w1 = ?1.50
w2 = 2.02

ii.Comment briefly on the effect of the ridge regularisation.

[2 marks]

The simple form of the equations makes obvious the “shrinkage” effect of the ridge penalty—it makes the weights smaller, shifting some variability from the model features into the noise.

[Total for Question 1: 25 marks]

2.The Karush-Kuhn-Tucker condition of complementary slackness requires that at the optimum w?
αigi(w?) = 0, ?i
where gi is an inequality constraint and αi is the corresponding Lagrange multiplier. What is the significance of this for support vector machines? (You may limit your answer to the hard margin case.)
[5 marks]

To fulfil this condition, at least one of αi, gi must always be zero. gi is only zero at equality, so αi must be zero otherwise. For SVMs, gi is the constraint that training sample xi be at or beyond the margin, so α is only non-zero for points exactly on the margin. But the solution is a weighted sum of the training samples (representer theorem), with the αi as weights:

w? = αixi
i

So only training samples on the margin—support vectors—contribute to the solution.

[Total for Question 2: 5 marks]

Figure 1: Operation graph for Question 3.

3.Figure 1 shows an operation graph representing a simplified neural network fragment that takes a single scalar input x1 and performs the following operations on it:

z = relu(w2 × relu(w1 × x1))

a.Given initial weights w1 = 0.4 and w2 = 3, perform a single stochastic gradient descent step for a data sample x1 = 6 with a learning rate of 0.1.
(You may use any convenient method to reproduce the figure for your answers: print, photograph, screen capture or sketch it by hand.)
i.Perform a forward pass through the network, writing the forward propagated data values in the shaded boxes above each arrow.
[2 marks]
See figure below. The values for these boxes are (left to right): 2.4, 2.4, 7.2 and 7.2.
[Question 3 cont. on next page]

[Question 3 cont.]

ii.Determine the local gradient at each node with respect to each of its inputs, entering the values in the boxes below the node. (Where there are two inputs, enter the gradient with respect to the upper input in the upper box and the gra-
dient with respect to the lower input in the lower box.)
[6 marks]
See figure in answer for part i. The values for these boxes are (left to right, top to bottom): 0.4, 6, 1, 3, 2.4 and 1.
iii.You are given that the final downstream loss gradient for the network is 1. Backpropagate this gradient through the graph, entering the values in the boxes below each arrow.
[6 marks]
See figure in answer for part i. The values for these boxes are (left to right, top to bottom): 1.2, 18, 3, 3, 1 and 1.

iv.Calculate the new values of w1 and w2.

[3 marks]

w1 = 0.4 ? (0.1 × 18) = ?1.4
w2 = 3 ? (0.1 × 2.4) = 2.76
[Question 3 cont. over page]

[Question 3 cont.]
b.What would have happened in part a if the initial value of w2 had been ?3 instead of 3?
[4 marks]
See figure below. The local gradient at the final relu would be 0, and during backpropagation this would zero out all the other loss gradients. The weights would remain unchanged.

c.Briefly discuss whether or not this is a problem.
[4 marks]
Not necessarily a problem—learning is iterative and individual weights need not be updated for every data point. And the situation is exaggerated here because the network is artifi- cially tiny, with only one input—in a real neural net each neuron would be integrating many inputs with many weights, which should produce more diverse responses. But it becomes a problem if it happens consistently—as it would in this case. Then this part of the network is “dead”: no further learning can occur and the output would be permanently 0.
Note there is an additional wrinkle in the very specific case presented here. With the final loss gradient given as constant 1, we are minimising the expression directly rather than a data-dependent loss function of it. In this case 0 really is the minimum, and so the dead neuron happens to produce the right answer. This would not generally be the case, though. (Gold star to the anonymous student who pointed this out.)

[Total for Question 3: 25 marks]

4.Decision trees are an example of a model class that is said to be “high variance”.

a.What does this mean?
[2 marks]
The model is highly sensitive to variations in the training data. Small changes may lead to significant differences in the trained model results, so it risks overfitting to random variations— noise—in the training set.

b.How do random forests attempt to mitigate this?
[2 marks]
Random forests build an ensemble of multiple trees using bootstrap resampling of the train- ing data and random subsets of the input features. Results are aggregated by averaging or voting, smoothing out the variance in exchange for potentially increased bias.

c.How is this similar to a popular approach to regularising neural networks?
[2 marks]
With dropout, different subsets of neurons are switched on and off for each training sample. Effectively, a different subnetwork is trained each time. At evaluation time, all neurons are on, aggregating the learning of all the random subsets.

[Total for Question 4: 6 marks]

5.Consider a convolutional neural network with the following layers:

?16 filters of size 3 × 3, applied at stride 1 with padding 1
?8 filters of size 5 × 5, applied at stride 1 with padding 2
?8 filters of size 3 × 3, applied at stride 1 with padding 1 If the input to the network is a 32 × 32 RGB image, then:
a.What size is the output from the network?

The general output size calculation for a conv layer is

[2 marks]

Dout

= Din ? K + 2P + 1
S

Here all layers have S = 1 and K = 2P + 1, so the spatial dimensions are unchanged throughout. The output depth is just the number of filters in the last layer, so the output size is 32 × 32 × 8.

b.What is the total number of parameters in the network? (Assume that all filters include a bias term.)
[3 marks]
Parameters per filter is w h d + 1. For layers beyond the first, d is the number of channels in the previous layer.
?layer 1: 16 × (3 × 3 × 3 + 1) = 16 × 28 = 448
?layer 2: 8 × (5 × 5 × 16 + 1) = 8 × 401 = 3208
?layer 3: 8 × (3 × 3 × 8 + 1) = 8 × 73 = 584
So total number of parameters is: 448 + 3208 + 584 = 4240

[Total for Question 5: 5 marks]

6.A simple hidden Markov model can be in two states, z1 and z2, with initial probabilities
π = h0.1 0.9i and transition probabilities
0.1 0.9
?0.0 1.0?
The model emits one of two symbols, p or q, at each time step. In state z1 the emission probabilities for p and q are 0.9 and 0.1, respectively, while in z2 they are 0.1 and 0.9. Given the observation sequence [p, p], what is the probability the model is in state z1 at timestep 2?
[5 marks]

z2 is absorbing, so the only possible state sequences are [z1, z1], [z1, z2] and [z2, z2]. In the absence of observations, the probabilities of each sequence would be:
P ([z1, z1]) = 0.1 × 0.1 = 0.01
P ([z1, z2]) = 0.1 × 0.9 = 0.09
P ([z2, z2]) = 0.9 × 1.0 = 0.90
Probability of the observed sequence for each state sequence is:
P ([p, p]|[z1, z1]) = 0.9 × 0.9 = 0.81
P ([p, p]|[z1, z2]) = 0.9 × 0.1 = 0.09
P ([p, p]|[z2, z2]) = 0.1 × 0.1 = 0.01
We must have observed one of these three scenarios, for which the overall probabilities are:
(0.01 × 0.81) + (0.09 × 0.09) + (0.90 × 0.01) = 0.0252
Only the first scenario has us in state z1 at step 2, so probability of that is:
(0.01 × 0.81)/0.0252 ≈ 0.32

[Total for Question 6: 5 marks]

7.What is the problem of “vanishing gradients” for neural networks? Why is this particu- larly relevant for vanilla recurrent neural networks?
[5 marks]

Neural nets are trained by gradient descent, and so require non-zero gradients in order to learn. Gradients are calculated using the chain rule, which involves multiplication of all the component gradients. If gradients are low, their product will get very small and become numerically unstable. If any gradient is zero, this will propagate back to all earlier neurons and learning will stall.
RNNs are especially susceptible because the same weights are shared across all steps in a sequence. During “backprop through time” these participate in the product repeatedly. This makes it difficult to learn long-range dependencies because gradients from many steps back will tend to vanish, leading to updates that are dominated by the most recent steps.

[Total for Question 7: 5 marks]

0.72 1.05 1.57 2.10 3.22 4.01
Table 2: Unlabelled sample data for Question 8

8.Table 2 shows a small dataset of unlabelled data with a single feature dimension.

a.Assign this data to two clusters using the k-Means algorithm.

i.Choose initial cluster centroid values, c1 and c2. Briefly justify your choice.
[2 marks]
Given the data is 1D and we’re dividing into two clusters, the two endpoints must be in different clusters, so it makes sense to start with c1 = 0.72, c2 = 4.01. But marks will be given for another choice if it comes with a decent argument.

ii.Iterate k-Means to convergence and give the final centroid estimates.
[4 marks]
?Initial cluster assignment is: [1, 1, 1, 1, 2, 2].
?Updated centroids are c1 = (0.72 + 1.05 + 1.57 + 2.10)/4 = 1.36, c2 = (3.22 + 4.01)/2 = 3.615.
?Updated cluster assignment is again [1, 1, 1, 1, 2, 2], so we’re done.

b.Do a single fitting iteration of a two-component Gaussian mixture model.
i.Use the results from part a to initialise the model. Use the centroid values c1, c2 as the initial means μ1, μ2. Choose initial values for the component probabilities α1, α2 and variances σ2, σ2 in some way. Explain your choices.
1 2
[4 marks]
Given cluster assignments, reasonable choices might be:
?α1 = 4/6 = 0.67
?α2 = 2/6 = 0.33
?σ2 = var(0.72, 1.05, 1.57, 2.10) = 0.27
?σ2 = var(3.22, 4.01) = 0.16
Again, other choices are fine if argued sensibly. Obviously we require α1 + α2 = 1

[Question 8 cont. over page]

[Question 8 cont.]
ii.Estimate the responsibilities γi,j of each component j for each sample i. Re- member to show your working.
[8 marks] Answers here depend on the choices above, so may vary a little between students— hence the reminder to show working—though probably not by very much unless the initial values are pretty wacky.
Responsibilities are calculated from the data values xi and model parameters as:
p(xi|μj, σ2)αj

γi,j = Σ

p(x |μ , σ2)α

Students may possibly calculate the pdf values explicitly from

1
p(x) = σ√2π exp

(x μ)2
— 2σ2

though more likely will use a library function such as scipy.stats.norm.pdf, which is allowed. Results from our initial values are:

Sample 1 2 3 4 5 6
Component 1 1.0 1.0 1..0 0.999 0.005 0.0
Component 2 0.0 0.0 0.0 0.001 0.995 1.0
iii.Use the responsibilities to update your estimates of μ1, μ2, σ2, σ2, α1 and α2.

Again, exact values may vary a bit according to initialisation. Means are updated as

1 2
[6 marks]

Variances as

μ = i γi,j xi i γi,j

σ2 =

Σi γi,j(xi ? μj)2

α = i γi,j
j n
With our responsibilities the updated values are:
?μ1 = 1.36
?μ2 = 3.62
?σ2 = 0.28
?σ2 = 0.16
?α1 = 0.67
?α2 = 0.33
(values are very marginally changed from the previous iteration, but the changes only reach the second decimal place for σ2)

[Total for Question 8: 24 marks]