Exegesis

Self-organised modular neural networks for encoding data

Posted on: Tuesday 1/1, 2008;  10:26 PM

These calculations are taken from the original Mathematica notebook used to calculate these results, which was started when I was working at the Isaac Newton Institute for Mathematical Sciences programme on Neural Networks in 1997. There are a few changes because of Mathematica version dependencies, but otherwise I have changed nothing. I calculate the "BE3408215191_1.gif" and "BE3408215191_2.gif" parts separately because that is the way I did the calculations originally.

This is the largest algebraic calculation that I have ever done. The results are sufficiently simple (and involve a great deal of algebraic cancellation) that one suspects that there might be a much cleverer way of deriving them. Anyone?

Currently, this reworking of the paper uses Mathematica version 5.

Preliminary Definitions

Switch off annoying warning messages.

"BE3408215191_3.gif"

Define a function for computing a Euclidean distance.

"BE3408215191_4.gif"

Define a function for contructing a vector on the unit circle centred at the origin.

"BE3408215191_5.gif"

Circular Manifold (2 overlapping posterior probabilities)

Functional Form of the Posterior Probability

Define p(θ)=f(θ) in the interval "BE3408215191_6.gif", where f(θ) is given by

"BE3408215191_7.gif"

Solve for the coefficients a, b and c by imposing continuity and normalisation.

"BE3408215191_8.gif"

"BE3408215191_9.gif"

Define a function to implement this solution.

"BE3408215191_10.gif"

Display f(θ) with the correct a, b and c inserted.

"BE3408215191_11.gif"

"BE3408215191_12.gif"

Stationary Posterior Probability

Without loss of generality, set y=0.

"BE3408215191_13.gif"

Compute the posterior average over reference vectors.

"BE3408215191_14.gif"

"BE3408215191_15.gif"

Compute the quantity that has to be zero when the posterior probability is stationary.

"BE3408215191_16.gif"

"BE3408215191_17.gif"

Insert the known form of the coefficients.

"BE3408215191_18.gif"

"BE3408215191_19.gif"

Solve for r.

"BE3408215191_20.gif"

"BE3408215191_21.gif"

Objective Function

Evaluate "BE3408215191_22.gif".

"BE3408215191_23.gif"

"BE3408215191_24.gif"

Evaluate "BE3408215191_25.gif".

"BE3408215191_26.gif"

"BE3408215191_27.gif"

Define a function to implement "BE3408215191_28.gif".

"BE3408215191_29.gif"

Stationary Reference Vector Length

Compute the quantity that has to be zero when the reference vector length is stationary.

"BE3408215191_30.gif"

"BE3408215191_31.gif"

Manipulate this into a simpler form.

"BE3408215191_32.gif"

"BE3408215191_33.gif"

Define a function to implement this.

"BE3408215191_34.gif"

Define a function to find a root of this.

"BE3408215191_35.gif"

Stationary "BE3408215191_36.gif"

Substitute in the known stationary parameter values to obtain the stationary "BE3408215191_37.gif".

"BE3408215191_38.gif"

"BE3408215191_39.gif"

Replace the "BE3408215191_40.gif" term from the stationarity condition on the reference vector length.

"BE3408215191_41.gif"

"BE3408215191_42.gif"

Define a function to implement this.

"BE3408215191_43.gif"

Define a function to minimise this w.r.t. s.

"BE3408215191_44.gif"

Toroidal Manifold (2 overlapping posterior probabilities)

Stationary Posterior Probability

Without loss of generality, set y=0.

"BE3408215191_45.gif"

Compute the posterior average over reference vectors.

"BE3408215191_46.gif"

"BE3408215191_47.gif"

Compute the quantity that has to be zero when the posterior probability is stationary.

"BE3408215191_48.gif"

"BE3408215191_49.gif"

Define a function for extracting the Fourier coefficients of an expression w.r.t. (cos 2θ,sin 2θ,cos θ,sin θ,1). This function assumes that these are the only Fourier components present.

"BE3408215191_50.gif"

Extract the Fourier coefficients w.r.t. (cos 2θ,sin 2θ,cos θ,sin θ,1).

"BE3408215191_51.gif"

Solve for r by setting the coefficient of cos 2θ to zero.

"BE3408215191_52.gif"

"BE3408215191_53.gif"

Verify that the solution makes all of the Fourier coefficients zero.

"BE3408215191_54.gif"

"BE3408215191_55.gif"

Objective Function

Evaluate "BE3408215191_56.gif".

"BE3408215191_57.gif"

"BE3408215191_58.gif"

Evaluate "BE3408215191_59.gif".

"BE3408215191_60.gif"

"BE3408215191_61.gif"

Define a function to implement "BE3408215191_62.gif".

"BE3408215191_63.gif"

Stationary Reference Vector Length

Compute the quantity that has to be zero when the reference vector length is stationary.

"BE3408215191_64.gif"

"BE3408215191_65.gif"

Manipulate this into a simpler form.

"BE3408215191_66.gif"

"BE3408215191_67.gif"

Define a function to implement this.

"BE3408215191_68.gif"

Define a function to find a root of this.

"BE3408215191_69.gif"

Stationary "BE3408215191_70.gif"

Substitute in the known stationary parameter values to obtain the stationarity "BE3408215191_71.gif".

"BE3408215191_72.gif"

"BE3408215191_73.gif"

Replace the "BE3408215191_74.gif" term from the stationarity condition on the reference vector length.

"BE3408215191_75.gif"

"BE3408215191_76.gif"

Define a function to implement this.

"BE3408215191_77.gif"

Define a function to minimise this w.r.t. s.

"BE3408215191_78.gif"

Circular Manifold (3 overlapping posterior probabilities)

Functional Form of the Posterior Probability

Define p(θ) for each of the intervals "BE3408215191_79.gif" (this is "BE3408215191_80.gif"), "BE3408215191_81.gif" (this is "BE3408215191_82.gif"), and "BE3408215191_83.gif" (this is "BE3408215191_84.gif").

"BE3408215191_85.gif"

State the continuity constraints.

"BE3408215191_86.gif"

"BE3408215191_87.gif"

State the norms, and expand them in terms of sin θ and cos θ.

"BE3408215191_88.gif"

"BE3408215191_89.gif"

Extract the oscillating parts of the norms.

"BE3408215191_90.gif"

"BE3408215191_91.gif"

Extract the constant parts of the norms.

"BE3408215191_92.gif"

"BE3408215191_93.gif"

Solve the continuity and normalisation constraints for the parameters {a2,a3,b1,b3,c1,c2,c3}, leaving a1 and b2 to be determined.

"BE3408215191_94.gif"

"BE3408215191_95.gif"

Define a function to implement this transformation.

"BE3408215191_96.gif"

Stationary Posterior Probability

Without loss of generality, set y=0.

Region 1: "BE3408215191_97.gif"

Compute the posterior average over reference vectors.

"BE3408215191_98.gif"

Compute the quantity that has to be zero when the posterior probability is stationary.

"BE3408215191_99.gif"

Extract the Fourier coefficients w.r.t. (cos 2θ,sin 2θ,cos θ,sin θ,1).

"BE3408215191_100.gif"

Region 2: "BE3408215191_101.gif"

Compute the posterior average over reference vectors.

"BE3408215191_102.gif"

Compute the quantity that has to be zero when the posterior probability is stationary.

"BE3408215191_103.gif"

Extract the Fourier coefficients w.r.t. (cos 2θ,sin 2θ,cos θ,sin θ,1).

"BE3408215191_104.gif"

Region 3: "BE3408215191_105.gif"

Compute the posterior average over reference vectors.

"BE3408215191_106.gif"

Compute the quantity that has to be zero when the posterior probability is stationary.

"BE3408215191_107.gif"

Extract the Fourier coefficients w.r.t. (cos 2θ,sin 2θ,cos θ,sin θ,1).

"BE3408215191_108.gif"

Solve the Stationarity Conditions:

The coefficient of cos(2θ) in region 2 gives b2.

"BE3408215191_109.gif"

"BE3408215191_110.gif"

Substitute back into the coefficient of cos θ in region 1.

"BE3408215191_111.gif"

"BE3408215191_112.gif"

Substitute back into the coefficient of sin θ in region 3.

"BE3408215191_113.gif"

"BE3408215191_114.gif"

Solve for a1.

"BE3408215191_115.gif"

"BE3408215191_116.gif"

Solve for r.

"BE3408215191_117.gif"

"BE3408215191_118.gif"

Substitute the solution back to determine the coefficients in the expression for the posterior probability.

"BE3408215191_119.gif"

"BE3408215191_120.gif"

Define a function to implement this transformation.

"BE3408215191_121.gif"

Display f1(θ) with the correct a1, b1 and c1 inserted.

"BE3408215191_122.gif"

"BE3408215191_123.gif"

Display f2(θ) with the correct a2, b2 and c2 inserted.

"BE3408215191_124.gif"

"BE3408215191_125.gif"

Display f3(θ) with the correct a3, b3 and c3 inserted.

"BE3408215191_126.gif"

"BE3408215191_127.gif"

Objective Function

Evaluate "BE3408215191_128.gif".

"BE3408215191_129.gif"

"BE3408215191_130.gif"

Evaluate "BE3408215191_131.gif".

"BE3408215191_132.gif"

"BE3408215191_133.gif"

Define a function to implement "BE3408215191_134.gif".

"BE3408215191_135.gif"

Stationary Reference Vector Length

Compute the quantity that has to be zero when the reference vector length is stationary.

"BE3408215191_136.gif"

"BE3408215191_137.gif"

Define a function to implement this.

"BE3408215191_138.gif"

Define a function to find a root of this.

"BE3408215191_139.gif"

Stationary "BE3408215191_140.gif"

Substitute in the known stationary parameter values to obtain the stationary "BE3408215191_141.gif".

"BE3408215191_142.gif"

Extract the main factor Last[d12circle3minimum] from "BE3408215191_143.gif", TrigFactor the sine terms, and write "BE3408215191_144.gif" as c2s.

"BE3408215191_145.gif"

"BE3408215191_146.gif"

Extract the main factor from the quantity that has to be zero when the reference vector length is stationary, and write "BE3408215191_147.gif" as c2s.

"BE3408215191_148.gif"

"BE3408215191_149.gif"

Solve for c2s in this stationarity condition.

"BE3408215191_150.gif"

"BE3408215191_151.gif"

Substitute back into the main factor in "BE3408215191_152.gif".

"BE3408215191_153.gif"

"BE3408215191_154.gif"

Include the missing factors that were dropped earlier.

"BE3408215191_155.gif"

"BE3408215191_156.gif"

Define a function to implement this.

"BE3408215191_157.gif"

Define a function to minimise this w.r.t. s.

"BE3408215191_158.gif"

Define a function which combines the 2 and 3 overlapping posterior probability cases.

"BE3408215191_159.gif"

Toroidal Manifold (3 overlapping posterior probabilities)

Stationary Posterior Probability

Without loss of generality, set y=0.

Region 1: "BE3408215191_160.gif"

Compute the posterior average over reference vectors.

"BE3408215191_161.gif"

Compute the quantity that has to be zero when the posterior probability is stationary.

"BE3408215191_162.gif"

Extract the Fourier coefficients w.r.t. (cos 2θ,sin 2θ,cos θ,sin θ,1).

"BE3408215191_163.gif"

Region 2: "BE3408215191_164.gif"

Compute the posterior average over reference vectors.

"BE3408215191_165.gif"

Compute the quantity that has to be zero when the posterior probability is stationary.

"BE3408215191_166.gif"

Extract the Fourier coefficients w.r.t. (cos 2θ,sin 2θ,cos θ,sin θ,1).

"BE3408215191_167.gif"

Region 3: "BE3408215191_168.gif"

Compute the posterior average over reference vectors.

"BE3408215191_169.gif"

Compute the quantity that has to be zero when the posterior probability is stationary.

"BE3408215191_170.gif"

Extract the Fourier coefficients w.r.t. (cos 2θ,sin 2θ,cos θ,sin θ,1).

"BE3408215191_171.gif"

Solve the Stationarity Conditions

The coefficient of the cos(2θ) term in region 2 gives b2.

"BE3408215191_172.gif"

"BE3408215191_173.gif"

Substitute back into the coefficient of cos θ in region 1.

"BE3408215191_174.gif"

"BE3408215191_175.gif"

Substitute back into the coefficient of sin θ in region 3.

"BE3408215191_176.gif"

"BE3408215191_177.gif"

Solve for a1.

"BE3408215191_178.gif"

"BE3408215191_179.gif"

Solve for r.

"BE3408215191_180.gif"

"BE3408215191_181.gif"

Substitute the solution back to determine the coefficients in the expression for the posterior probability.

"BE3408215191_182.gif"

"BE3408215191_183.gif"

Define a function to implement this transformation.

"BE3408215191_184.gif"

Display f1(θ) with the correct a1, b1 and c1 inserted.

"BE3408215191_185.gif"

"BE3408215191_186.gif"

Display f2(θ) with the correct a2, b2 and c2 inserted.

"BE3408215191_187.gif"

"BE3408215191_188.gif"

Display f3(θ) with the correct a3, b3 and c3 inserted.

"BE3408215191_189.gif"

"BE3408215191_190.gif"

The functional form of the toroidal posterior probability is exactly the same as in the circular case, except that "BE3408215191_191.gif".

Objective Function

Evaluate "BE3408215191_192.gif".

"BE3408215191_193.gif"

"BE3408215191_194.gif"

Evaluate "BE3408215191_195.gif".

"BE3408215191_196.gif"

"BE3408215191_197.gif"

Define a function to implement "BE3408215191_198.gif".

"BE3408215191_199.gif"

Stationary Reference Vector Length

Compute the quantity that has to be zero when the reference vector length is stationary.

"BE3408215191_200.gif"

"BE3408215191_201.gif"

Define a function to implement this.

"BE3408215191_202.gif"

Define a function to find a root of this.

"BE3408215191_203.gif"

Stationary "BE3408215191_204.gif"

Substitute in the known stationary parameter values to obtain the stationary "BE3408215191_205.gif".

"BE3408215191_206.gif"

"BE3408215191_207.gif"

Extract the main factor Last[d12torus3factorialminimum] from "BE3408215191_208.gif", TrigFactor the sine term, and write "BE3408215191_209.gif" as cs1cs2s.

"BE3408215191_210.gif"

"BE3408215191_211.gif"

Extract the main factor from the quantity that has to be zero when the reference vector length is stationary, and write "BE3408215191_212.gif" as cs1cs2s.

"BE3408215191_213.gif"

"BE3408215191_214.gif"

Solve for cs1cs2s in this stationarity condition.

"BE3408215191_215.gif"

Substitute back into the main factor in "BE3408215191_216.gif".

"BE3408215191_217.gif"

"BE3408215191_218.gif"

Include the missing factors that were dropped earlier.

"BE3408215191_219.gif"

"BE3408215191_220.gif"

Define a function to implement this.

"BE3408215191_221.gif"

Define a function to minimise this w.r.t. s.

"BE3408215191_222.gif"

Define a function which combines the 2 and 3 overlapping posterior probability cases.

"BE3408215191_223.gif"

Collect Results Together

These results are as computed originally using Mathematica version 3. They are algebraically equivalent to whatever result is obtained using the latest version of Mathematica.

Miscellaneous

"BE3408215191_224.gif"

"BE3408215191_225.gif"

In the torus routine the number of neurons M is automatically divided by 2 inside the routine.

"BE3408215191_226.gif"

2 Overlapping Posterior Probabilities

"BE3408215191_227.gif"

"BE3408215191_228.gif"

"BE3408215191_229.gif"

"BE3408215191_230.gif"

"BE3408215191_231.gif"

"BE3408215191_232.gif"

"BE3408215191_233.gif"

"BE3408215191_234.gif"

"BE3408215191_235.gif"

"BE3408215191_236.gif"

"BE3408215191_237.gif"

"BE3408215191_238.gif"

"BE3408215191_239.gif"

"BE3408215191_240.gif"

"BE3408215191_241.gif"

"BE3408215191_242.gif"

"BE3408215191_243.gif"

"BE3408215191_244.gif"

3 Overlapping Posterior Probabilities

"BE3408215191_245.gif"

"BE3408215191_246.gif"

"BE3408215191_247.gif"

"BE3408215191_248.gif"

"BE3408215191_249.gif"

"BE3408215191_250.gif"

"BE3408215191_251.gif"

"BE3408215191_252.gif"

"BE3408215191_253.gif"

"BE3408215191_254.gif"

"BE3408215191_255.gif"

"BE3408215191_256.gif"

"BE3408215191_257.gif"

"BE3408215191_258.gif"

"BE3408215191_259.gif"

The torus results below assume that the total number of neurons is M, and this is automatically divided by 2 inside the routines. These results are redundant because the functional form of the toroidal posterior probability is exactly the same as in the circular case, except that "BE3408215191_260.gif".

"BE3408215191_261.gif"

"BE3408215191_262.gif"

2/3 Overlapping Posterior Probabilities

"BE3408215191_263.gif"

"BE3408215191_264.gif"

"BE3408215191_265.gif"

"BE3408215191_266.gif"

Asymptotic Properties

Circular Manifold

Validity Limit

Derive the limiting form (as n→∞ and M→∞) of the boundary between the regions where 2 or 3 posterior probabilities overlap.

"BE3408215191_267.gif"

"BE3408215191_268.gif"

M→∞ (2 overlapping posterior probabilities)

Do an asymptotic expansion of s in the quantity that must be zero when s is stationary.

"BE3408215191_269.gif"

"BE3408215191_270.gif"

Solve for the coefficients that satisfy stationarity.

"BE3408215191_271.gif"

"BE3408215191_272.gif"

Substitute back into "BE3408215191_273.gif" and asymptotically expand to lowest contributing order.

"BE3408215191_274.gif"

"BE3408215191_275.gif"

Verify this result graphically. The plots lie on top of each other.

"BE3408215191_276.gif"

"BE3408215191_277.gif"

n→∞ (3 overlapping posterior probabilities)

Do an expansion of s about the point "BE3408215191_278.gif" (which is the limiting value as n→∞). Expand s as "BE3408215191_279.gif", where δs is presumed to be O(1).

"BE3408215191_280.gif"

"BE3408215191_281.gif"

This stationarity condition is thus satisfied to "BE3408215191_282.gif" provided that the first and third terms (which are both O(1)) cancel out.

"BE3408215191_283.gif"

"BE3408215191_284.gif"

Substitute the solution for δs back into "BE3408215191_285.gif".

"BE3408215191_286.gif"

"BE3408215191_287.gif"

Extract the leading asymptotic behaviour.

"BE3408215191_288.gif"

"BE3408215191_289.gif"

Verify this result graphically. The plots lie on top of each other.

"BE3408215191_290.gif"

"BE3408215191_291.gif"

Toroidal Manifold (joint encoding case)

Validity Limit

Derive the limiting form (as n→∞ and M→∞) of the boundary between the regions where 2 or 3 posterior probabilities overlap.

"BE3408215191_292.gif"

"BE3408215191_293.gif"

Toroidal Manifold (factorial encoding case)

Validity Limit

Derive the limiting form (as n→∞ and M→∞) of the boundary between the regions where 2 or 3 posterior probabilities overlap.

"BE3408215191_294.gif"

"BE3408215191_295.gif"

M→∞ (2 overlapping posterior probabilities)

Do an asymptotic expansion of s in the quantity that must be zero when s is stationary.

"BE3408215191_296.gif"

"BE3408215191_297.gif"

Solve for the coefficients that satisfy stationarity.

"BE3408215191_298.gif"

"BE3408215191_299.gif"

Substitute back into "BE3408215191_300.gif" and asymptotically expand to lowest contributing order.

"BE3408215191_301.gif"

"BE3408215191_302.gif"

Verify this result graphically. The plots lie on top of each other.

"BE3408215191_303.gif"

"BE3408215191_304.gif"

n→∞ (3 overlapping posterior probabilities)

Do an expansion of s about the point "BE3408215191_305.gif" (which is the limiting value as n→∞). Adjust the expansion so that it is of the form "BE3408215191_306.gif", where δs is presumed to be O(1).

"BE3408215191_307.gif"

"BE3408215191_308.gif"

This stationarity condition is thus satisfied to "BE3408215191_309.gif" provided that the first and third terms (which are both O(1)) cancel out.

"BE3408215191_310.gif"

"BE3408215191_311.gif"

Substitute the solution for δs back into "BE3408215191_312.gif".

"BE3408215191_313.gif"

"BE3408215191_314.gif"

Manipulate the large factor separately, then recombine the factors, then extract the leading asymptotic behaviour.

"BE3408215191_315.gif"

"BE3408215191_316.gif"

Verify this result graphically. The plots lie on top of each other.

"BE3408215191_317.gif"

"BE3408215191_318.gif"

Joint/Factorial Encoding Stability Limit

The n→∞ behaviour of "BE3408215191_319.gif" in the joint encoding case is "BE3408215191_320.gif", and in the factorial encoding case is "BE3408215191_321.gif".

Solve for the value of M that makes these the same.

"BE3408215191_322.gif"

"BE3408215191_323.gif"

Assorted Results

Value of n on Validity Boundary

For what value of n does "BE3408215191_324.gif" in the circular case with 2 overlapping posterior probabilities?

"BE3408215191_325.gif"

"BE3408215191_326.gif"

Check this solves the case of 3 overlapping posterior probabilities.

"BE3408215191_327.gif"

"BE3408215191_328.gif"

Verify that the distortion functions join up correctly.

"BE3408215191_329.gif"

"BE3408215191_330.gif"

For what value of n does "BE3408215191_331.gif" in the toroidal case (factorial encoding) with 2 overlapping posterior probabilities?

"BE3408215191_332.gif"

"BE3408215191_333.gif"

Check this solves the case of 3 overlapping posterior probabilities.

"BE3408215191_334.gif"

"BE3408215191_335.gif"

Verify that the distortion functions join up correctly.

"BE3408215191_336.gif"

"BE3408215191_337.gif"

Verify that Solutions Join Together

The difference between the 2 and 3 overlapping posterior probabilities "BE3408215191_338.gif" at the boundary "BE3408215191_339.gif" is a multiple of a quantity that is zero.

"BE3408215191_340.gif"

"BE3408215191_341.gif"

Ditto in the toroidal case when "BE3408215191_342.gif".

"BE3408215191_343.gif"

"BE3408215191_344.gif"

The 2 overlapping posterior probabilities circle and torus solutions are closely related.

"BE3408215191_345.gif"

"BE3408215191_346.gif"

Plots

The functional forms were computed using Mathematica version 3, and are algebraically equivalent to whatever the current version of Mathematica gives.

Receptive Field Plots (Circular Manifold)

Define a function which plots the receptive fields.

"BE3408215191_347.gif"

Plot an example of the receptive fields and their norm for encoding a circle in the case of 2 overlapping posterior probabilities.

"BE3408215191_348.gif"

"BE3408215191_349.gif"

Plot an example of the receptive fields and their norm for encoding a circle in the case of 3 overlapping posterior probabilities.

"BE3408215191_350.gif"

"BE3408215191_351.gif"

Stability Contour Plots (Toroidal Manifold)

"BE3408215191_352.gif"

Define an area of the (n,M) plane to investigate.

"BE3408215191_353.gif"

Plot the asymptotic value of M.

"BE3408215191_354.gif"

"BE3408215191_355.gif"

Plot the boundary between the 2 and 3 overlapping posterior probability regions for joint encoding.

"BE3408215191_356.gif"

"BE3408215191_357.gif"

Plot the boundary between the 2 and 3 overlapping posterior probability regions for factorial encoding.

"BE3408215191_358.gif"

"BE3408215191_359.gif"

Derive some interpolating functions to speed up the computation of "BE3408215191_360.gif".

"BE3408215191_361.gif"

"BE3408215191_362.gif"

"BE3408215191_363.gif"

"BE3408215191_364.gif"

Plot the boundary between the stable regions for joint and factorial encoding.

"BE3408215191_365.gif"

"BE3408215191_366.gif"

Combine the plots.

"BE3408215191_367.gif"

"BE3408215191_368.gif"

"BE3408215191_369.gif"

Do a comparative plot of "BE3408215191_370.gif" for the joint and factorial encoding cases.

"BE3408215191_371.gif"

"BE3408215191_372.gif"

"BE3408215191_373.gif"

Manually Simplified Expressions

These expressions are obtained by starting with the results produced by Mathematica 3 (or whatever algebraically equivalent result is obtained using the current version of Mathematica) and interactively simplifying them.

scircle2[M,n,s]

"BE3408215191_374.gif"

"BE3408215191_375.gif"

"BE3408215191_376.gif"

"BE3408215191_377.gif"

"BE3408215191_378.gif"

"BE3408215191_379.gif"

scircle3[M,n,s]

"BE3408215191_380.gif"

"BE3408215191_381.gif"

Do the following replacements to get all of the trigonometric s-dependence in terms of "BE3408215191_382.gif" and "BE3408215191_383.gif".

"BE3408215191_384.gif"

"BE3408215191_385.gif"

Manually transform.

"BE3408215191_386.gif"

"BE3408215191_387.gif"

"BE3408215191_388.gif"

"BE3408215191_389.gif"

Manually transform some more.

"BE3408215191_390.gif"

"BE3408215191_391.gif"

"BE3408215191_392.gif"

"BE3408215191_393.gif"

storus2factorial[M,n,s]

"BE3408215191_394.gif"

"BE3408215191_395.gif"

"BE3408215191_396.gif"

"BE3408215191_397.gif"

"BE3408215191_398.gif"

"BE3408215191_399.gif"

storus3factorial[M,n,s]

"BE3408215191_400.gif"

"BE3408215191_401.gif"

Transform in terms of "BE3408215191_402.gif", "BE3408215191_403.gif" and "BE3408215191_404.gif".

"BE3408215191_405.gif"

"BE3408215191_406.gif"

Manually transform into a nice form.

"BE3408215191_407.gif"

"BE3408215191_408.gif"

"BE3408215191_409.gif"

"BE3408215191_410.gif"

Manually transform some more.

"BE3408215191_411.gif"

"BE3408215191_412.gif"

"BE3408215191_413.gif"

"BE3408215191_414.gif"

dcircle2[M,n,s]

"BE3408215191_415.gif"

"BE3408215191_416.gif"

"BE3408215191_417.gif"

"BE3408215191_418.gif"

"BE3408215191_419.gif"

"BE3408215191_420.gif"

dcircle3[M,n,s]

"BE3408215191_421.gif"

"BE3408215191_422.gif"

Use that "BE3408215191_423.gif".

"BE3408215191_424.gif"

"BE3408215191_425.gif"

"BE3408215191_426.gif"

dtorus2factorial[M,n,s]

"BE3408215191_427.gif"

"BE3408215191_428.gif"

"BE3408215191_429.gif"

"BE3408215191_430.gif"

"BE3408215191_431.gif"

"BE3408215191_432.gif"

dtorus3factorial[M,n,s]

"BE3408215191_433.gif"

"BE3408215191_434.gif"

Use that "BE3408215191_435.gif".

"BE3408215191_436.gif"

"BE3408215191_437.gif"

"BE3408215191_438.gif"

Some Useful Diagrams

"BE3408215191_439.gif"

Target Manifolds

Define 1D image manifolds for 1 and 2 (superimposing) targets.

"BE3408215191_440.gif"

Define 2D image manifold for 1 target.

"BE3408215191_441.gif"

Plot 3 pixel values of a 1D 1-target image manifold.

"BE3408215191_442.gif"

"BE3408215191_443.gif"

Plot 3 pixel values of a 1D 2-target image manifold.

"BE3408215191_444.gif"

"BE3408215191_445.gif"

Plot 3 pixel values of a 2D 1-target image manifold.

"BE3408215191_446.gif"

"BE3408215191_447.gif"

2 Overlapping Posterior Probabilities

"BE3408215191_448.gif"

"BE3408215191_449.gif"
"BE3408215191_450.gif"
"BE3408215191_451.gif"

3 Overlapping Posterior probabilities

"BE3408215191_452.gif"

"BE3408215191_453.gif"
"BE3408215191_454.gif"
"BE3408215191_455.gif"

Torus Pictures

Pinch some torus drawing code from Graphics`Shapes`. There must be a better way of doing it than this, but I need to be able to "get at" the torus drawing code in order to be able to draw things on the surface of the torus.

"BE3408215191_456.gif"

"BE3408215191_457.gif"

"BE3408215191_458.gif"

"BE3408215191_459.gif"

"BE3408215191_460.gif"

"BE3408215191_461.gif"

Add some extra routines for drawing things on the surface of a torus.

"BE3408215191_462.gif"

"BE3408215191_463.gif"

"BE3408215191_464.gif"

"BE3408215191_465.gif"

Draw some toruses.

"BE3408215191_466.gif"

"BE3408215191_467.gif"

"BE3408215191_468.gif"

"BE3408215191_469.gif"

"BE3408215191_470.gif"

"BE3408215191_471.gif"

"BE3408215191_472.gif"

"BE3408215191_473.gif"

Gather together some of the graphics.

"BE3408215191_474.gif"

"BE3408215191_475.gif"

Valid Regions

"BE3408215191_476.gif"

Graphics:None

Joint Posterior Probabilities (linear case)

Here is a function for plotting the boundaries between the piecewise linear regions of a 2D posterior probability.

"BE3408215191_478.gif"

Plot the case "BE3408215191_479.gif".

"BE3408215191_480.gif"

"BE3408215191_481.gif"

Define the posterior probability centred on the origin.

"BE3408215191_482.gif"

Evaluate the integral of the posterior probability over all values of its input vector.

"BE3408215191_483.gif"

"BE3408215191_484.gif"

Plot the posterior probability.

"BE3408215191_485.gif"

"BE3408215191_486.gif"

Plot the marginalised posterior probability. This demonstrates that you obtain the 1D posterior probability when you marginalise the 2D case.

"BE3408215191_487.gif"

"BE3408215191_488.gif"

Permalink   Notebook

A self-organising approach to multiple classifier fusion

Posted on: Tuesday 1/1, 2008;  3:01 PM

These results are generated using the same Mathematica code that was used for the original paper. A few changes had to be made to make the code compatible with the latest version of Mathematica, and to make it more consistent and easy to read.

I have used the previously trained network(s) to generate the figures. I should supply code to generate new instances of the figures.

Currently, this reworking of the paper uses Mathematica version 5.

Functions

Definitions of functions for computing SVQ posterior probabilities.

"BE3408188510_1.gif"

"BE3408188510_2.gif"

"BE3408188510_3.gif"

Figure 1

Define parameters of an SVQ network trained on data from a circular manifold.

"BE3408188510_4.gif"

"BE3408188510_5.gif"

This is figure 1.

"BE3408188510_6.gif"

"BE3408188510_7.gif"

Figures 2 and 3

Target output.

"BE3408188510_8.gif"

"BE3408188510_9.gif"

Strong supervision case. Define parameters of an SVQ network trained on data from a toroidal manifold with strong supervision from a  target as given above.

"BE3408188510_10.gif"

"BE3408188510_11.gif"

"BE3408188510_12.gif"

"BE3408188510_13.gif"

"BE3408188510_14.gif"

"BE3408188510_15.gif"

Weak supervision case. Define parameters of an SVQ network trained on data from a toroidal manifold with weak supervision from a  target as given above.

"BE3408188510_16.gif"

"BE3408188510_17.gif"

"BE3408188510_18.gif"

"BE3408188510_19.gif"

"BE3408188510_20.gif"

"BE3408188510_21.gif"

Unsupervised joint encoder case. Define parameters of an SVQ network trained in the joint encoding regime on data from a toroidal manifold.

"BE3408188510_22.gif"

"BE3408188510_23.gif"

"BE3408188510_24.gif"

"BE3408188510_25.gif"

Unsupervised factorial encoder case. Define parameters of an SVQ network trained in the factorial encoding regime on data from a toroidal manifold.

"BE3408188510_26.gif"

"BE3408188510_27.gif"

"BE3408188510_28.gif"

"BE3408188510_29.gif"

This is figure 2.

"BE3408188510_30.gif"

"BE3408188510_31.gif"

This is figure 3.

"BE3408188510_32.gif"

"BE3408188510_33.gif"

Permalink   Notebook

A discrete firing event analysis of the adaptive cluster expansion network

Posted on: Tuesday 1/1, 2008;  2:46 PM

This paper contains only diagrams. I have written some Mathematica code to generate a similar set of pictures. The code is not optimised to make full use of reusable components.

Currently, this reworking of the paper uses Mathematica version 5.

Initialisation

Turn off annoying error messages.

"BE3408187610_1.gif"

Functions

"BE3408187610_2.gif"

"BE3408187610_3.gif"

"BE3408187610_4.gif"

Figure 1

"BE3408187610_5.gif"

"BE3408187610_6.gif"

"BE3408187610_7.gif"

"BE3408187610_8.gif"

Figure 2

"BE3408187610_9.gif"

"BE3408187610_10.gif"

"BE3408187610_11.gif"

"BE3408187610_12.gif"

Figure 3

"BE3408187610_13.gif"

"BE3408187610_14.gif"

Permalink   Notebook

Self-organisation of multiple winner-take-all neural networks

Posted on: Tuesday 1/1, 2008;  2:14 PM

These results are created from the original Mathematica notebook that I used to create the paper. Some minor modifications had to be made to ensure compatibility with the latest version of Mathematica.

Currently, this reworking of the paper is being converted so that it uses Mathematica version 6.

Initialisation

Turn off annoying warning messages.

"BE3408185654_1.gif"

Define some fonts for use in the graphics.

"BE3408185654_2.gif"

Load packages.

"BE3408185654_3.gif"

Functions

Borrow some Mathematica code for building a torus from the Graphics`Shapes` package.

"BE3408185654_4.gif"

"BE3408185654_5.gif"

"BE3408185654_6.gif"

Define various bands encircling a torus.

"BE3408185654_7.gif"

"BE3408185654_8.gif"

"BE3408185654_9.gif"

Define a torus plus various bands around it.

"BE3408185654_10.gif"

"BE3408185654_11.gif"

"BE3408185654_12.gif"

"BE3408185654_13.gif"

Define some functions to plot various graphs.

"BE3408185654_14.gif"

"BE3408185654_15.gif"

Figure 1

Assorted useful torus graphics.

"BE3408185654_16.gif"

"BE3408185654_17.gif"

"BE3408185654_18.gif"

"BE3408185654_19.gif"

"BE3408185654_20.gif"

"BE3408185654_21.gif"

"BE3408185654_22.gif"

"BE3408185654_23.gif"

This is figure 1.

"BE3408185654_24.gif"

"BE3408185654_25.gif"

Figure 2

This is figure 2.

"BE3408185654_26.gif"

"BE3408185654_27.gif"

Figure 3

Create the various pieces of graphics needed to construct figure 3.

"BE3408185654_28.gif"

"BE3408185654_29.gif"

"BE3408185654_30.gif"

"BE3408185654_31.gif"

This is figure 3.

"BE3408185654_32.gif"

"BE3408185654_33.gif"

Figure 4

This is figure 4.

"BE3408185654_34.gif"

"BE3408185654_35.gif"

Figure 5

This is figure 5.

"BE3408185654_36.gif"

"BE3408185654_37.gif"

Figure 6

Create the various pieces of graphics needed to construct figure 6.

"BE3408185654_38.gif"

"BE3408185654_39.gif"

"BE3408185654_40.gif"

"BE3408185654_41.gif"

"BE3408185654_42.gif"

This is figure 6.

"BE3408185654_43.gif"

"BE3408185654_44.gif"

Figure 7

This is figure 7.

"BE3408185654_45.gif"

"BE3408185654_46.gif"

Figure 8

This is figure 8.

"BE3408185654_47.gif"

"BE3408185654_48.gif"

Figure 9

This is figure 9.

"BE3408185654_49.gif"

"BE3408185654_50.gif"

Figure 10

Create the various pieces of graphics needed to construct figure 10.

"BE3408185654_51.gif"

Graphics:None

"BE3408185654_53.gif"

This is figure 10.

"BE3408185654_54.gif"

Graphics:None

Permalink   Notebook

Prior knowledge and object reconstruction using the best linear estimate technique

Posted on: Monday 31/12, 2007;  6:44 PM

Reproduction of the computational results in "Prior knowledge and object reconstruction using the best linear estimate technique", Luttrell S P, Optica Acta, 1985, vol. 32, no. 6, pp. 703-716.

I can reproduce most of the results in the paper. However, the paper has gaps in it where things have not been defined explicitly. I presume this happened because bits of the paper got deleted at a last minute editorial stage, and this left some dangling bonds in the paper. This leaves undefined the precise meaning of "object basis function" which could be either "BE3408115476_1.gif" (defined) or "BE3408115476_2.gif" (undefined).

Currently, this reworking of the paper uses Mathematica version 5.

Initialisation

Turn off the warning messages that occur because of the infinite integration range. This problem could be avoided by doing the W=constant part of the integral analytically.

"BE3408115476_3.gif"

Turn off warning messages for similar symbol names.

"BE3408115476_4.gif"

Load the package for multiple plots.

"BE3408115476_5.gif"

Load the package for displaying plots together.

"BE3408115476_6.gif"

Functions

Define the sinc function.

"BE3408115476_7.gif"

Define a single matrix element of the kernel in equation 24. I could evaluate the contribution from the W=constant part of the integral analytically.

"BE3408115476_8.gif"

Define the full matrix kernel assuming samples spaces at Δ times the Nyquist length centred on the origin. The matrix is symmetric but I don't bother using this fact when computing it.

"BE3408115476_9.gif"

Define the object basis function as in equation 25.

"BE3408115476_10.gif"

Table 1

Compute the eigenvalues of the matrix kernel for 25 Nyquist samples with "BE3408115476_11.gif"and c=π, and a unit half-width top-hat weight W. The "BE3408115476_12.gif"multiplying the eigenvalues is the sampling density factor.

"BE3408115476_13.gif"

"BE3408115476_14.gif"

Save the results because they took a while to compute.

"BE3408115476_15.gif"

"BE3408115476_16.gif"

List the first 7 eigenvalues for "BE3408115476_17.gif"and c=π. This is table 1.

"BE3408115476_18.gif"

"BE3408115476_19.gif" c=π
0.783368 0.981041
0.196959 0.736059
0.0113728 0.24359
0.000190051 0.0219907
2.15127*10^^-6 0.00106412
1.08678*10^^-8 0.0000220308
5.81144*10^^-11 4.72969*10^^-7

Figure 1

Compute the eigenvalues of the matrix kernel for 25 Nyquist samples (times various sampling interval spacing parameters Δ={1,0.5,0.25,0.1}) with c=π, and a unit half-width top-hat weight W sitting on a pedestal ε.

"BE3408115476_20.gif"

"BE3408115476_21.gif"

Save the results because they took a while to compute.

"BE3408115476_22.gif"

"BE3408115476_23.gif"

Plot the eigenvalues. This is figure 1.

"BE3408115476_24.gif"

Graphics:None

Figure 2

Compute the eigenvalues of the matrix kernel for 25 Nyquist samples with c=π and "BE3408115476_26.gif", and a unit half-width top-hat weight W sitting on a various pedestals ε={0.1,0.01,0.0001}.

"BE3408115476_27.gif"

"BE3408115476_28.gif"

Save the results because they took a while to compute.

"BE3408115476_29.gif"

"BE3408115476_30.gif"

Plot the eigenvalues. This is figure 2.

"BE3408115476_31.gif"

Graphics:None

Figure 3

Compute the eigenvalues of the matrix kernel for 25 Nyquist samples with c=π and "BE3408115476_33.gif", and a Gaussian weight "BE3408115476_34.gif" sitting on a various pedestals ε={0.1,0.01,0.0001}.

"BE3408115476_35.gif"

"BE3408115476_36.gif"

Save the results because they took a while to compute.

"BE3408115476_37.gif"

"BE3408115476_38.gif"

Plot the eigenvalues. This is figure 3.

"BE3408115476_39.gif"

Graphics:None

These results are different from figure 3 for low order eigenvalues. It seems that I used the wrong normalisation (or something) for the Gaussian in the paper. An easy way to see that the paper is wrong is that the sum of the eigenvalues is not large enough: there is too much depression of the low order eigenvalues which is not compensated enough by the increase in high order eigenvalues.

Figure 4

Compute the eigenvectors of the matrix kernel for 25 Nyquist samples with c=π, and a unit half-width top-hat weight W sitting on a pedestal ε=0.1.

"BE3408115476_41.gif"

"BE3408115476_42.gif"

Save the results because they took a while to compute.

"BE3408115476_43.gif"

"BE3408115476_44.gif"

Plot the normalised object basis functions. This is figure 4.

"BE3408115476_45.gif"

"BE3408115476_46.gif"

The function "BE3408115476_47.gif" (rather than "BE3408115476_48.gif") was originally plotted, but I didn't define "BE3408115476_49.gif" in the paper; it probably was removed during a last minute editorial change. I have found "BE3408115476_50.gif" defined in an early draft (July 1983) of the paper.

Figure 5

Compute the eigenvectors of the matrix kernel for 25 Nyquist samples with c=π, and a Gaussian weight "BE3408115476_51.gif" sitting on a pedestal ε=0.1.

"BE3408115476_52.gif"

"BE3408115476_53.gif"

Save the results because they took a while to compute.

"BE3408115476_54.gif"

"BE3408115476_55.gif"

Plot the normalised object basis functions. This is figure 5.

"BE3408115476_56.gif"

"BE3408115476_57.gif"

The normalisations of these results are different from figure 5.

Figure 6

This makes use of the results already computed for figure 5.

Compute the corresponding results for ε=0.01.

"BE3408115476_58.gif"

"BE3408115476_59.gif"

Save the results because they took a while to compute.

"BE3408115476_60.gif"

"BE3408115476_61.gif"

Plot the normalised object basis functions for ε=0.1 and ε=0.01. This is figure 6.

"BE3408115476_62.gif"

"BE3408115476_63.gif"

The normalisation(s) differ from figure 6 but otherwise this plot seems to be correct. I have labelled the function plotted as "BE3408115476_64.gif" (rather than "BE3408115476_65.gif" as it was labelled in the paper).

Figure 7

Compute resolution enhancement factor E as ratio of half-width at half-height of (continuous) sinc image to its reconstruction from 25 Nyquist (spaced for c=π) samples for unit half-wifth weight sitting on a pedestal "BE3408115476_66.gif" and using various values of c. Note that the sample spacing stays fixed throughout  at Nyquist for c=π, and only the bandwidth of the imaging function is varied.

"BE3408115476_67.gif"

"BE3408115476_68.gif"

Save the results because they took a while to compute.

"BE3408115476_69.gif"

"BE3408115476_70.gif"

Plot the resolution enhancement factor. This is figure 7.

"BE3408115476_71.gif"

Graphics:None

This accurately matches figure 7 apart from the "BE3408115476_73.gif" case which I assume is due to rounding errors. There is one other problem point for c=1.5 and "BE3408115476_74.gif" The behaviour at small c looks as if it is due to the half-width at half-height of the reconstruction falling on the fixed-location discontinuity of the top-hat W.

Figure 8

The same as figure 7 except using Gaussian weight "BE3408115476_75.gif" sitting on a pedestal.

"BE3408115476_76.gif"

"BE3408115476_77.gif"

Save the results because they took a while to compute.

"BE3408115476_78.gif"

"BE3408115476_79.gif"

Plot the resolution enhancement factor. This is figure 8.

"BE3408115476_80.gif"

Graphics:None

This accurately matches figure 8 apart from the "BE3408115476_82.gif" case which I assume is due to rounding errors.

Try to find out how I got wrong results in Gaussian case. I have a vague memory that I discretised object space which would give inaccurate results.

Tidy up the errors in the resolution enhancement plots in figures 7 and 8.

Permalink   Notebook

Recent Posts
1/1, 2008; 10:26 PM:
Self-organised modular neural networks for encoding data
1/1, 2008; 3:01 PM:
A self-organising approach to multiple classifier fusion
1/1, 2008; 2:46 PM:
A discrete firing event analysis of the adaptive cluster expansion network
1/1, 2008; 2:14 PM:
Self-organisation of multiple winner-take-all neural networks
31/12, 2007; 6:44 PM:
Prior knowledge and object reconstruction using the best linear estimate technique

Archive


Links

 

Blogged from
A WorkLife FrameWork by
Scientific Arts

All material on this website Copyright © 2008, Stephen Luttrell.