From: I.N. Galidakis on
Tim Little wrote:

> On 2009-09-02, I.N. Galidakis <morpheus(a)olympus.mons> wrote:
>> As far as I can see, the reconstruction works always, up to a
>> bottleneck digit of the original message.
>
> The point is, how do you know whether a given digit is a bottleneck
> digit of the original message without having the original message? In
> my example, the initial digit was indistinguishable from a bottleneck
> digit with the information available to a reconstruction algorithm.

The obvious: You know when you are in a bottleneck point of the original
message, when part of the original message shows in LC(L).

> For a case very similar to your example, suppose your original data had
> differed by just one digit (and been slightly longer). When encoded:
>
> 9515098625262515281009078659738611668767861522527602764253927456709
> 1542712484889165100931555067.

That's not a correct encoding. I copied your sequence above to Maple:

>
L:=[9,5,1,5,0,9,8,6,2,5,2,6,2,5,1,5,2,8,1,0,0,9,0,7,8,6,5,9,7,3,8,6,1,1,6,6,8,7,
6,7,
8,6,1,5,2,2,5,2,7,6,0,2,7,6,4,2,5,3,9,2,7,4,5,6,7,0,9,1,5,4,2,7,1,2,4,8,4,8,8,9,
1,6,
5,1,0,0,9,3,1,5,5,5,0,6,7];

and it gives:

> LC(L);
[9, 5, 1, 5, 0, 9, 8, 6, 2, 5, 2, 6, 2, *4*, *0*, *9*, 8, 1, 0, 0, 9]

The digits starting at position 14 are all messed-up, so your message is not
encoded correctly.

> This is truncated at the same position, to:
>
> 15281009078659738611668767861522527602764253927456709
> 1542712484889165100931555067.
>
> You recover the prefix
>
> 150986252625|152810...
>
> But now the next block to the left (as the procedure requires) is
> "25". It does not start with a bottleneck digit of the original
> message, but you have no way to know which reconstructed digits are
> bottleneck digits. So a spurious "2" is prefixed to the sequence and
> the reconstruction goes wrong.
>
>
>> You might want to try with a string of length 40-50 or better, with
>> sub-messages starting at bottleneck digits.
>
> There are sequences of arbitrary length for which the error still
> holds. I merely chose a short one for ease of demonstration.
>
>
> - Tim
--
Ioannis

From: Tim Little on
On 2009-09-03, I.N. Galidakis <morpheus(a)olympus.mons> wrote:
>> 9515098625262515281009078659738611668767861522527602764253927456709
>> 1542712484889165100931555067.
>
> That's not a correct encoding.

A simple typo. Apparently when I constructed the sequence, I typed
"45" when I intended to hit "5".


Here's a much more straightforward example of why reconstruction must
sometimes fail. The following three sequences are valid fixed points:

72543212754731450939128121787733950542405677765323421441656735,
352543212754731450939128121787733950542405677765323421441656735,
442543212754731450939128121787733950542405677765323421441656735.

They can all be truncated at a bottleneck digit to the same sequence:

2754731450939128121787733950542405677765323421441656735.

No reconstruction algorithm can determine which of the three was
originally intended, because all distinguishing information has been
lost. (There are actually more than three possibilities; others not
listed here for brevity)


What's more, the number of valid reconstructions grows very rapidly
with the number of truncated digits, regardless of the length of the
rest of the message. Longer messages merely give more overlap with
already known digits.


- Tim
From: I.N. Galidakis on
Tim Little wrote:

> On 2009-09-03, I.N. Galidakis <morpheus(a)olympus.mons> wrote:
>>> 9515098625262515281009078659738611668767861522527602764253927456709
>>> 1542712484889165100931555067.
>>
>> That's not a correct encoding.
>
> A simple typo. Apparently when I constructed the sequence, I typed
> "45" when I intended to hit "5".
>
>
> Here's a much more straightforward example of why reconstruction must
> sometimes fail.

Read "sometimes" when "there are not enough digits in L".

> The following three sequences are valid fixed points:


> 72543212754731450939128121787733950542405677765323421441656735,
> 352543212754731450939128121787733950542405677765323421441656735,
> 442543212754731450939128121787733950542405677765323421441656735.

An LC "fixed point" is not a finite string. It's an infinite string of digits.
All the examples we have been discussing are just "segments" of fixed points. If
I put your streams in my Maple generator to generate more digits (so that more
digits are encoded), the initial segments are predicted correctly:

For the first one:

> L:=[7,2,5,4,3,2,1,2,7,5,4,7,3,1,4,5,0,9,3,9,1,2,8,1,2,
1,7,8,7,7,3,3,9,5,0,5,4,2,4,0,5,6,7,7,7,6,5,3,2,3,4,2,
1,4,4,1,6,5,6,7,3,5];

> G:=GFPLC(L,30);#encode 30 more blocks

G := [7, 2, 5, 4, 3, 2, 1, 2, 7, 5, 4, 7, 3, 1, 4, 5, 0, 9, 3, 9, 1, 2, 8, 1, 2,
1, 7, 8, 7, 7, 3, 3, 9, 5, 0, 5, 4, 2, 4, 0, 5, 6, 7, 7, 7, 6, 5, 3, 2, 3, 4, 2,
1, 4, 4, 1, 6, 5, 6, 7, 3, 5, 0, 9, 5, 5, 6, 6, 6, 2, 7, 1, 3, 4, 5, 9, 6, 7, 5,
6, 0, 2, 0, 8, 1, 2, 5, 8, 4, 9, 0, 9, 4, 6, 7, 1, 2, 6, 1, 7, 0, 8, 0, 3, 6, 1,
8, 9, 9, 7, 0, 0, 6, 1, 7, 0, 0, 8, 7, 1, 7, 7, 0, 8, 4, 1, 8, 7, 3, 5, 7, 3, 0,
9, 9, 3, 1, 7, 5, 7, 5, 7, 3, 5, 9, 3, 3, 0, 0, 5, 0, 6, 9, 6, 4, 4, 7, 2, 2, 4,
4, 3, 4, 3, 0, 5, 7, 9, 4, 6, 6, 8, 5, 9, 1, 2, 7, 0, 0, 7, 1, 2, 9, 7, 3, 5, 7,
0, 5, 6, 7, 1, 2, 4, 3, 1, 7, 6, 6, 5, 3, 4, 9]

Now if I truncate at your fixed point:

> L := [2, 7, 5, 4, 7, 3, 1, 4, 5, 0, 9, 3, 9, 1, 2, 8, 1, 2, 1, 7, 8, 7, 7, 3,
3, 9, 5, 0, 5, 4, 2, 4, 0, 5, 6, 7, 7, 7, 6, 5, 3, 2, 3, 4, 2, 1, 4, 4, 1, 6, 5,
6, 7, 3, 5, 0, 9, 5, 5, 6, 6, 6, 2, 7, 1, 3, 4, 5, 9, 6, 7, 5, 6, 0, 2, 0, 8, 1,
2, 5, 8, 4, 9, 0, 9, 4, 6, 7, 1, 2, 6, 1, 7, 0, 8, 0, 3, 6, 1, 8, 9, 9, 7, 0, 0,
6, 1, 7, 0, 0, 8, 7, 1, 7, 7, 0, 8, 4, 1, 8, 7, 3, 5, 7, 3, 0, 9, 9, 3, 1, 7, 5,
7, 5, 7, 3, 5, 9, 3, 3, 0, 0, 5, 0, 6, 9, 6, 4, 4, 7, 2, 2, 4, 4, 3, 4, 3, 0, 5,
7, 9, 4, 6, 6, 8, 5, 9, 1, 2, 7, 0, 0, 7, 1, 2, 9, 7, 3, 5, 7, 0, 5, 6, 7, 1, 2,
4, 3, 1, 7, 6, 6, 5, 3, 4, 9]

> LC(L);
[2, 5, 4, 3, 2, 1, |, 2, 7, 5, 4, 7, 3, 1, 4, 5, 0, 9, 3, 9, 1, 2, 8, 1, 2, 1,
7, 8, 7, 7, 3, 3, 9, 5, 0, 5, 4, 2, 4, 0, 5, 6, 7, 7, 7, 6] (predicted left of
"|")

It follows that the bottleneck point is 7, so the message matches your first
stream:

[7, 2, 5, 4, 3, 2, 1, 2, 7, 5, 4, 7, 3, 1, 4, 5, 0, 9, 3, 9, 1, 2, 8, 1, 2, 1,
7, 8, 7, 7, 3, 3, 9, 5, 0, 5, 4, 2, 4, 0, 5, 6, 7, 7, 7, 6]

If I put your second example in the generator:

> L:=[3,5,2,5,4,3,2,1,2,7,5,4,7,3,1,4,5,0,9,3,9,1,2,8,
1,2,1,7,8,7,7,3,3,9,5,0,5,4,2,4,0,5,6,7,7,7,6,5,3,2,
3,4,2,1,4,4,1,6,5,6,7,3,5];

> G:=GFPLC(L,30);#encode 30 more blocks:

G := [3, 5, 2, 5, 4, 3, 2, 1, 2, 7, 5, 4, 7, 3, 1, 4, 5, 0, 9, 3, 9, 1, 2, 8, 1,
2, 1, 7, 8, 7, 7, 3, 3, 9, 5, 0, 5, 4, 2, 4, 0, 5, 6, 7, 7, 7, 6, 5, 3, 2, 3, 4,
2, 1, 4, 4, 1, 6, 5, 6, 7, 3, 5, 0, 9, 6, 2, 4, 8, 4, 5, 2, 7, 3, 0, 0, 9, 1, 1,
5, 3, 8, 5, 7, 6, 1, 2, 5, 8, 7, 1, 9, 3, 9, 2, 0, 1, 2, 7, 1, 7, 8, 0, 8, 5, 3,
8, 8, 2, 2, 3, 7, 5, 5, 0, 7, 2, 7, 8, 7, 7, 3, 7, 7, 2, 0, 0, 5, 1, 3, 9, 4, 3,
5, 5, 9, 2, 1, 3, 3, 7, 1, 1, 2, 5, 3, 0, 6, 3, 0, 5, 5, 4, 4, 9, 4, 5, 9, 9, 2,
4, 4, 8, 1, 3, 0, 5, 4, 2, 8, 0, 6, 7, 5, 2, 9, 1, 7, 5, 6, 6, 2, 7, 6, 7, 0, 0,
8, 4, 0, 1, 7, 7, 0, 8, 1, 8, 7, 6, 9, 0, 2, 6, 5]

Now if I truncate at your fixed point:

L := [2, 7, 5, 4, 7, 3, 1, 4, 5, 0, 9, 3, 9, 1, 2, 8, 1, 2, 1, 7, 8, 7, 7, 3, 3,
9, 5, 0, 5, 4, 2, 4, 0, 5, 6, 7, 7, 7, 6, 5, 3, 2, 3, 4, 2, 1, 4, 4, 1, 6, 5, 6,
7, 3, 5, 0, 9, 6, 2, 4, 8, 4, 5, 2, 7, 3, 0, 0, 9, 1, 1, 5, 3, 8, 5, 7, 6, 1, 2,
5, 8, 7, 1, 9, 3, 9, 2, 0, 1, 2, 7, 1, 7, 8, 0, 8, 5, 3, 8, 8, 2, 2, 3, 7, 5, 5,
0, 7, 2, 7, 8, 7, 7, 3, 7, 7, 2, 0, 0, 5, 1, 3, 9, 4, 3, 5, 5, 9, 2, 1, 3, 3, 7,
1, 1, 2, 5, 3, 0, 6, 3, 0, 5, 5, 4, 4, 9, 4, 5, 9, 9, 2, 4, 4, 8, 1, 3, 0, 5, 4,
2, 8, 0, 6, 7, 5, 2, 9, 1, 7, 5, 6, 6, 2, 7, 6, 7, 0, 0, 8, 4, 0, 1, 7, 7, 0, 8,
1, 8, 7, 6, 9, 0, 2, 6, 5]

> LC(L);

[2, 5, 4, 3, 2, 1, |, 2, 7, 5, 4, 7, 3, 1, 4, 5, 0, 9, 3, 9, 1, 2, 8, 1,
2, 1, 7, 8, 7, 7, 3, 3, 9, 5, 0, 5, 4, 2, 4, 0, 5, 6, 7, 7, 7, 6] (predicted
left of "|")

Now put the block: 5,4,3,2,1 and update:

> L := [5, 4, 3, 2, 1, 2, 7, 5, 4, 7, 3, 1, 4, 5, 0, 9, 3, 9, 1, 2, 8, 1, 2, 1,
7, 8, 7, 7, 3, 3, 9, 5, 0, 5, 4, 2, 4, 0, 5, 6, 7, 7, 7, 6, 5, 3, 2, 3, 4, 2, 1,
4, 4, 1, 6, 5, 6, 7, 3, 5, 0, 9, 6, 2, 4, 8, 4, 5, 2, 7, 3, 0, 0, 9, 1, 1, 5, 3,
8, 5, 7, 6, 1, 2, 5, 8, 7, 1, 9, 3, 9, 2, 0, 1, 2, 7, 1, 7, 8, 0, 8, 5, 3, 8, 8,
2, 2, 3, 7, 5, 5, 0, 7, 2, 7, 8, 7, 7, 3, 7, 7, 2, 0, 0, 5, 1, 3, 9, 4, 3, 5, 5,
9, 2, 1, 3, 3, 7, 1, 1, 2, 5, 3, 0, 6, 3, 0, 5, 5, 4, 4, 9, 4, 5, 9, 9, 2, 4, 4,
8, 1, 3, 0, 5, 4, 2, 8, 0, 6, 7, 5, 2, 9, 1, 7, 5, 6, 6, 2, 7, 6, 7, 0, 0, 8, 4,
0, 1, 7, 7, 0, 8, 1, 8, 7, 6, 9, 0, 2, 6, 5]

>LC(L);

[5, 2, |, 5, 4, 3, 2, 1, 2, 7, 5, 4, 7, 3, 1, 4, 5, 0, 9, 3, 9, 1, 2, 8,
1, 2, 1, 7, 8, 7, 7, 3, 3, 9, 5, 0, 5, 4, 2, 4, 0, 5, 6, 7, 7, 7,6] (predicted
left of "|")

It now follows that the first digit is 3, and the complete message is:


[3, 5, 2, 5, 4, 3, 2, 1, 2, 7, 5, 4, 7, 3, 1, 4, 5, 0, 9, 3, 9, 1, 2,
8, 1, 2, 1, 7, 8, 7, 7, 3, 3, 9, 5, 0, 5, 4, 2, 4, 0, 5, 6, 7, 7, 7, 6]

which matches your second stream. Similar for the third stream.

> They can all be truncated at a bottleneck digit to the same sequence:
>
> 2754731450939128121787733950542405677765323421441656735.
>
> No reconstruction algorithm can determine which of the three was
> originally intended, because all distinguishing information has been
> lost. (There are actually more than three possibilities; others not
> listed here for brevity)
>
>
> What's more, the number of valid reconstructions grows very rapidly
> with the number of truncated digits, regardless of the length of the
> rest of the message. Longer messages merely give more overlap with
> already known digits.

The validity of the reconstruction depends on the length of L. If we have
"enough" encoded digits in the received message, the message's beginning can be
reconstructed completely.

The question is, how "many" encoded digits are enough for a correct
reconstruction. The answer is of course: "infinite", since a "fixed point" of LC
is an infinite stream of digits. Since it's of course impossible for practical
purposes to send an infinite stream, any error in the reconstruction will sooner
or later show up, after enough digits come in.

This depends on the ability of the transmitter to keep sending encoded digits,
even after the reconstruction completes on the part of the receiver. If, after a
certain number of additional sent digits the reconstruction changes, this means
that the receiver's reconstruction was not correct.

> - Tim
--
Ioannis

From: I.N. Galidakis on
I.N. Galidakis wrote:
[snip]

> Similar for the third stream.

Here's how the third stream can be done:

> L:=[4,4,2,5,4,3,2,1,2,7,5,4,7,3,1,4,5,0,9,3,9,1,2,8,1,2,1,7,8,
7,7,3,3,9,5,0,5,4,2,4,0,5,6,7,7,7,6,5,3,2,3,4,2,1,4,4,1,6,5,6,7,3,5];

> G:=GFPLC(L,40);

G := [4, 4, 2, 5, 4, 3, 2, 1, 2, 7, 5, 4, 7, 3, 1, 4, 5, 0, 9, 3, 9, 1, 2, 8, 1,
2, 1, 7, 8, 7, 7, 3, 3, 9, 5, 0, 5, 4, 2, 4, 0, 5, 6, 7, 7, 7, 6, 5, 3, 2, 3, 4,
2, 1, 4, 4, 1, 6, 5, 6, 7, 3, 5, 0, 9, 1, 0, 7, 3, 6, 8, 5, 8, 3, 1, 9, 9, 5, 3,
7, 0, 4, 5, 6, 1, 1, 2, 7, 8, 9, 6, 8, 6, 9, 3, 1, 1, 2, 4, 1, 7, 6, 3, 2, 1, 3,
0, 8, 1, 8, 3, 0, 4, 5, 1, 7, 5, 4, 9, 0, 2, 2, 7, 3, 7, 9, 5, 1, 8, 3, 3, 3, 3,
5, 3, 9, 9, 7, 7, 9, 4, 8, 5, 5, 5, 7, 1, 2, 7, 0, 5, 8, 2, 7, 8, 4, 7, 7, 4, 2,
5, 4, 7, 2, 9, 0, 5, 2, 4, 5, 5, 6, 8, 8, 6, 5, 4, 7, 3, 0, 9, 1, 9, 7, 7, 3, 5,
4, 1, 6, 9, 7, 1, 6, 5, 3, 4, 3, 6, 9, 6, 0, 6, 0, 5, 5, 5, 6, 6, 3, 6, 2, 2, 7,
3, 1, 4, 4, 5, 6, 7, 2, 5, 1, 4, 6, 0, 2, 4, 0, 8, 5, 1]

Now I pick up the truncated:

> L := [2, 7, 5, 4, 7, 3, 1, 4, 5, 0, 9, 3, 9, 1, 2, 8, 1, 2, 1, 7, 8, 7, 7, 3,
3, 9, 5, 0, 5, 4, 2, 4, 0, 5, 6, 7, 7, 7, 6, 5, 3, 2, 3, 4, 2, 1, 4, 4, 1, 6, 5,
6, 7, 3, 5, 0, 9, 1, 0, 7, 3, 6, 8, 5, 8, 3, 1, 9, 9, 5, 3, 7, 0, 4, 5, 6, 1, 1,
2, 7, 8, 9, 6, 8, 6, 9, 3, 1, 1, 2, 4, 1, 7, 6, 3, 2, 1, 3, 0, 8, 1, 8, 3, 0, 4,
5, 1, 7, 5, 4, 9, 0, 2, 2, 7, 3, 7, 9, 5, 1, 8, 3, 3, 3, 3, 5, 3, 9, 9, 7, 7, 9,
4, 8, 5, 5, 5, 7, 1, 2, 7, 0, 5, 8, 2, 7, 8, 4, 7, 7, 4, 2, 5, 4, 7, 2, 9, 0, 5,
2, 4, 5, 5, 6, 8, 8, 6, 5, 4, 7, 3, 0, 9, 1, 9, 7, 7, 3, 5, 4, 1, 6, 9, 7, 1, 6,
5, 3, 4, 3, 6, 9, 6, 0, 6, 0, 5, 5, 5, 6, 6, 3, 6, 2, 2, 7, 3, 1, 4, 4, 5, 6, 7,
2, 5, 1, 4, 6, 0, 2, 4, 0, 8, 5, 1]

> LC(L);

[2, 5, 4, 3, 2, 1, |, 2, 7, 5, 4, 7, 3, 1, 4, 5, 0, 9, 3, 9, 1, 2, 8, 1, 2, 1,
7, 8, 7, 7, 3, 3, 9, 5, 0, 5, 4, 2, 4, 0, 5, 6, 7, 7, 7, 6, 5, 3, 2, 3, 4, 2, 1,
4, 4, 1] (predicted digits left of "|")

Pick up the block 4,3,2,1, add it to the list and update:

L := [4, 3, 2, 1, 2, 7, 5, 4, 7, 3, 1, 4, 5, 0, 9, 3, 9, 1, 2, 8, 1, 2, 1, 7, 8,
7, 7, 3, 3, 9, 5, 0, 5, 4, 2, 4, 0, 5, 6, 7, 7, 7, 6, 5, 3, 2, 3, 4, 2, 1, 4, 4,
1, 6, 5, 6, 7, 3, 5, 0, 9, 1, 0, 7, 3, 6, 8, 5, 8, 3, 1, 9, 9, 5, 3, 7, 0, 4, 5,
6, 1, 1, 2, 7, 8, 9, 6, 8, 6, 9, 3, 1, 1, 2, 4, 1, 7, 6, 3, 2, 1, 3, 0, 8, 1, 8,
3, 0, 4, 5, 1, 7, 5, 4, 9, 0, 2, 2, 7, 3, 7, 9, 5, 1, 8, 3, 3, 3, 3, 5, 3, 9, 9,
7, 7, 9, 4, 8, 5, 5, 5, 7, 1, 2, 7, 0, 5, 8, 2, 7, 8, 4, 7, 7, 4, 2, 5, 4, 7, 2,
9, 0, 5, 2, 4, 5, 5, 6, 8, 8, 6, 5, 4, 7, 3, 0, 9, 1, 9, 7, 7, 3, 5, 4, 1, 6, 9,
7, 1, 6, 5, 3, 4, 3, 6, 9, 6, 0, 6, 0, 5, 5, 5, 6, 6, 3, 6, 2, 2, 7, 3, 1, 4, 4,
5, 6, 7, 2, 5, 1, 4, 6, 0, 2, 4, 0, 8, 5, 1]

> LC(L);

[4, 2, 5, 4, 3, 2, 1, 2, 7, 5, 4, 7, 3, 1, 4, 5, 0, 9, 3, 9, 1, 2, 8, 1, 2, 1,
7, 8, 7, 7, 3, 3, 9, 5, 0, 5, 4, 2, 4, 0, 5, 6, 7, 7, 7, 6, 5, 3, 2, 3, 4, 2, 1,
4, 4, 1]

from which it follows that the first bottleneck digit is 4, hence:

[4, 4, 2, 5, 4, 3, 2, 1, 2, 7, 5, 4, 7, 3, 1, 4, 5, 0, 9, 3, 9, 1, 2, 8, 1, 2,
1, 7, 8, 7, 7, 3, 3, 9, 5, 0, 5, 4, 2, 4, 0, 5, 6, 7, 7, 7, 6, 5, 3, 2, 3, 4, 2,
1, 4, 4, 1]

which is your third stream. But I see what you are saying on this case:

The truncated:

> L:=[2,7,5,4,3,2,1,2,7,5,4,7,3,1,4,5,0,9,3,9,1,2,8,1,2,1,7,8,
7,7,3,3,9,5,0,5,4,2,4,0,5,6,7,7,7,6,5,3,2,3,4,2,1,4,4,1,6,5,6,7,3,5];

has problems, because it can predict different sequences. The problem stems from
the fact that:

> LC(L);
[2, 5, 2, 5, 4, 3, 2, 1,|,2, 7, 5, 4, 7, 3, 1, 4, 5] (predicted digits before
"|")

contains 5 successive bottlenecks prior to the stream 2,7,5,4,3,1,4,5.

Here, I can pick up just the digit (1), the two digits (2,1), the three digits
(3,2,1), the four digits (4,3,2,1), the five digits (5,4,3,2,1) or the digits
(7,2,5,2,5,4,3,2,1) and continue the algorithm to give me different predictions
with an updated stream.

Obviously such cases cannot predict a unique sequence, since there's no way to
choose the next block uniquely. However, I think it is fairly obvious that each
combination choice determines a unique sequence and all choices can be
enumerated easily (as was demonstrated with your three example streams), so all
possible message decodings are available to the receiver to choose from.

[snip]
--
Ioannis

From: Tim Little on
On 2009-09-04, I.N. Galidakis <morpheus(a)olympus.mons> wrote:
> If I put your streams in my Maple generator to generate more digits
> (so that more digits are encoded), the initial segments are
> predicted correctly:

They were not predicted at all. LC(L) was identical in each case, yet
you chose different ways to process the data because you knew what the
result ought to be. That's not an algorithm, and cannot be made to be
one.

Having pointed out the same property in two different ways without
success, I wish you good luck in analyzing your system by yourself.


- Tim