Late reply because I was constructing a wooden cage to hold a ceiling fan since the house I rent is so poorly designed that there are no wall nor ceiling studs capable of supporting a ceiling fan. So I lost most of the day being a carpenter without power tools.The container is very efficiently seek-able over the network, in fact. But your implementation must be sufficiently intelligent.
Here are some benchmarks from the opusfile library, On a 25 hour long variable bitrate audio recording performing 1000 jumps to exact sample positions with no caching:
Total seek operations: 1873 (1.873 per exact seek, 4 maximum).
So an average of less than two operations, and in the worst case 4-- meaning even with a 100ms RTT a worse case seek will not cause a noticeable delay. (and obviously, if it cached, it would be much better).
I am not an expert on streaming media and have just begun my research, but it seems to me that your quoted benchmark is assuming many seeks will be done on the stream. But there are cases where the user wants to only skip once or twice into a song as they are sampling the music for the first time, which is my use case. For that case, the lack of an index is afaics horrific because there will be the latency of some roundtrips required to optimally locate the seek position (by a bisection sampling method) and wasted bandwidth as well.
No, it is assuming the user will make exactly one seek. Otherwise, it would assume caching. The expected accesses given are what you would get for a user that makes only a single seek.
I can't find the details for this benchmark any where on the Opus website. I presume you are expecting that I would have to download, install, and build from source and then study the source code to understand what the benchmark is actually measuring. That seems to be an inappropriate level of work for an interested person to do when they simply want to get a reasonable first-pass analysis of the performance issues.
As you know, the devil is in the details. I don't know for example if the benchmark is using some heuristics to reduce the error for estimation of the file position for seeks given it has been given 1000 trials. Also where you wrote "jumps to exact sample positions", I don't know if you mean the jumps are to exact positions in time and thus estimations of file position. Also I don't know the frame size that was employed (i.e. I know Opus goes as low as 20ms but this increases the overhead) and whether that is a significant factor. I think I read that an entire page much be downloaded before the checksum can be computed, to be sure one doesn't confuse garbled data with a frame header.
I presume the error in estimation would increase if the sound content was highly variable in complexity and thus the VBR encoding would generate highly variable file positions relative to desired seek position in time. I am wondering if the 4 seeks worst case would hold. What if it climbed to 8 seeks!
I didn't check how large the page size is, so I don't know if bandwidth wasted is a significant factor with erroneous seeks in the file position.
In any case, one certain error in your analysis appears to be that you presume the internet can be modelled with a mean latency (e.g. 100ms). The internet I presume has hiccups and I presume the latency is Gaussian distributed. Thus 1 or 2 extra round-trips (e.g. 100ms each direction ideally, and worse from the Davao, Mindanao where I am) could be the difference between an unnoticeable delay of for example 500ms or a noticeable delay of 1+ seconds. And the occasional 4+ seeks, could mean up to seconds of delay. What may be more compelling is the extra load on the server, the hard disks thrashing, and also the latency of the server on each seek.
Perhaps this has already all been studied exhaustively, but unfortunately I can't find that data with Google, so it appears to me to be another set of risks. Adopters have to factor in risks in their assessment.
You could have instead made the index optional.
This is also not a free choice. In existing formats with optional indexes _no_ commonly used software is able to correctly seek when the "optional" index is not present. Many have no support for that case at all at all, and what has support in theory doesn't work due to inadequate testing.
This was a similar argument made by the IETF group working on the Websocket framing protocol when I argued they should make the framing orthogonal to the data layer. I quit because they were not going to change their mind and I presume they did not adopt my proposal.
I think it is not a valid argument because many implementations may not correctly implement the more obtuse seeking without an index. Thus we get the worst of both worlds.
I just don't buy this notion that we should dumb down and make inferior standards, because people are lazy to write and test software properly. Geez did Vint Cerf follow that theology when he design TCP/IP? Did we conflate all the network layers and collapse them into one layer because the OSI model is complex. Hell no! The End-to-End Principle is fundamental!
Doing so would also increase the overhead for the format by 20% or so. As mentioned, accurate indexes are not small-- and many things compromise by just not providing accurate indexes; which then leaves applications linearly scanning or not permitting sample accurate seeking.
I assume the 20% estimate is only for when the optional index is present. So it is presumed someone would use an index only when that 20% was justified by their use case. Again I argue you should not remove degrees-of-freedom and hinder the optimization of use cases which you did not envision because no group or person is omniscient.
And how is not having the index any worse than not allowing an index. I fail to see the logic. Seems you are arguing that the receiving end will expect indexes and not be prepared for the case where indexes are not present. But that is a bug in the receiving end's software then. And in that case, there is no assurance that software would have done the index-less seeking more efficiently for the status quo of not allowing an index. None of this makes sense to me.
Also I don't understand how you calculate 20% increase in file size for adding an index. For example, lets take an average 180 second song consuming roughly 5MB for VBR encoding. Let's assume my users are satisfied with seeking in 1 second increments, so that means means I need at most 180 of 22-bit indices, so that is only 495 bytes which is only a 0.01% increase! On top of that I could even compress those 22-bit indices into relative offsets if I want to shrink it by roughly 75% to 0.0025%.
Seems you are excluding use cases.
What use case is excluded?
I have alluded to scenarios in this post. Permute them. And there are use cases that neither of us are aware of. We are not omniscient. We should not top-down remove degrees-of-freedom. Some of the people designing Web standards these days are doing it wrong and are not of the same pedigree as Vint Cerf. I butted heads with them in the past (including Ian Hickson!), and I am tired of arguing with you guys. Just do what ever you want, I am going to route around the failure.
Moreover-- what error is there in having a streaming container that is actually good at streaming? People can use other containers if they need other properties.
Because it is very difficult to get new standards widely promulgated. The cost of not having Opus supported well for use cases. I will probably end up using Opus and having to roll my own client side instead of using HTML5 (which means my stuff will work best only on Android and not in the browser and probably not as well on iOS at least at the start). So I am very thankful for the work you did on Opus if I go that route. But I just think you should acknowledge to yourself that you are really awesome at your core competency, but you math guys need to allow guys like myself to participate in the standardization process for high-level issues. I have worked for example on what is now Corel Painter (the natural media paint tool not Corel Paint). Meaning I have dealt with the masses (Painter-J shipped a million units in the 1990s for example).
Here is
an example of where I promised Mozilla they were shooting themselves in the foot and they obstinately refused to listen, even after dozens of developers complained as I promised Mozilla would happen.
(Though ironically, the best supported non-streaming container that is widely used with Opus-- MKV-- which has 'optional' indexes, on average requires transferring more data to seek a single time than Opus in Ogg does; due to the need to transfer the whole index to seek only once). The height of flexibility is acceptance that not every goal can be optimally satisfied in a single design.
Perhaps their index is very high resolution, very lengthy stream, and/or there is no option to break the index up so not the entire index is put at the start of the file (all of which could be fixed with a better protocol design and specification). I provided my computation above for a 495 byte index.
Hey I have absolutely no experience at all with streaming. I am doing this all in my head for the first time. I think that exemplifies that I am an experienced coder and designer.
Hey I understand what it is like to deal with trolls and in that case I would support the swift action since trolls only aim to disrupt, but I wasn't trolling you.
I have no clue what you're going on about here. I responded politely to your message here-- are you referring to my non-amusement to your wall of insults, allegations, and personal attacks that you cross-posted to several threads; when you couldn't find your own post when it had been moved to offtopic? I apologize if my abrasive response makes you feel threatened enough to need to expound on your best racist theories; but I have had several of our best posters in the past tell me they stopped using Bitcoin Talk because of harassment by you in particular, and its certainly a situation I've felt myself. You may not intend to troll; but your radical jumps into insults and attacks combined with your persistence and the stream of consciousness diving into incoherent technobabble arguments which you sometimes present is effectively indistinguishable from trolling. I and others have expended a lot of time trying to encourage better behavior under your now-banned past identities, without success. All that is left is to say that if you are unable to behave yourself and not act like a troll, you will be banned. Otherwise you won't be. The choice is yours.
My
reply is here.
Gregory I worked for Tom Hedges on Painter and he could memorize a page of the phone book by looking at it for few moments. His IQ was off the charts. He would send me 1 sentence directives and it was up to me to figure out what was all reachable from that one sentence.
What may seem incoherent to you is perhaps because you are not investing the time to follow links and the discussion that has taken place.
It is inane to suggest I should re-type everything I already typed over and over again just so that no one has to follow a link.
Actually Gregory, several times I have felt I respect you and admire your work and even presentations. Obviously I do, because I want to use some of your work. And I've learned important things from you. I have already explained
what seems to rub me the wrong way. I don't know how to fix that, other than to stop interacting.
Btw I think I did actually figure out how to remove the proof-of-square in CCT (but perhaps I have a math error but I have looked it several times and can't find an error but it just seems too simple and I don't know how you guys could have missed it, so there must be an error). I will release the white paper where I had combined CCT with CN rings back in June/July. But I want to reorganize the paper a bit based on some insights I gained from the discussions with Shen-noether.
As for the politics, never mind (insoluble). People will think what ever they think. I have my justifications and reasons, but it isn't worth trying to convince others of my side. It is time for head in the sand, coding. Thanks. Talk is cheap as you know.
