Today’s update focuses on the work completed from July 15th through July 20th.
Much of the work completed this week focused around deploying upgrades to Oyster Storage’s backend. One of these upgrades included optimizing the backup script that currently creates emergency backups of the data maps presently stored on Oyster’s private tangle. When attempting these upgrades, the problems arose when trying to move recent changes into Oyster’s primary database. The Go framework the development team is using includes a migration tool called Fizz which developers can use to write files which will make changes to a database. Unfortunately, Fizz is not sophisticated enough to refrain from trying to create a table if it already exists or refrains from adding a column if it already exists, which caused errors anytime the development team attempted to pull changes from the master branch and then run migrations on the production machines. Thus, the development team needed to rewrite the migrations in SQL, so that the team could check to see if a table already exists and subsequently not create a duplicate of that table, as well as not adding a column to a table if it already exists. The team had to go through several iterations of SQL migrations before the file would work on both the developer broker databases and the Aurora DBs. Oyster’s development machines use MariaDB Docker images, while the production machines use AWS Aurora DBs with MySQL 5.7 server engines, and an early iteration of SQL migrations worked on the docker image databases but not on Aurora; thus, these SQL migrations had to be re-written. These changes led to SQL migration files that would work on both the production and development machines.
Furthermore, upon attempting to perform the migrations on the production databases, it became apparent that the size of the completed_data_maps table (which stores the uploaded user data submitted via Oyster Storage) would prevent the migrations from running successfully because of the size of the table. The table is very large because the developers did not yet delete old completed data maps and because of an indexing error which allowed duplicates into the table. The team is currently working to remove duplicates from this table, which will enable the team to run migrations, subsequently leading to the production brokers being brought back online.
For the time being, the development team will be bringing an additional set of production brokers online so that Oyster Storage can remain functional while these SQL migrations are completed on the master production infrastructure. Also, our community should expect to see a significant increase in file upload size by mid next week.
In addition to the work mentioned above, the team has continued working on integrating the oyster-streamable package into the web interface. This change will allow more memory efficient file upload and downloads, which means Oyster will be able to support larger file sizes. This is in the final stages of QA and testing, and so the team is hoping to deploy this change soon. Additionally, the team has been cleaning up the oyster-streamable package such that it can be released as a software development kit (SDK) for the public to use Oyster’s storage protocol. Finally, further work was completed on installing TypeScript support in both the web node and web interface, and several bugs and issues regarding the Badger DB deployment have been fixed.
