Artificial intelligence wrote a total dictation for the first time — five failed due to “hallucinations”

The Scribe system showed better results than the developers expected from it

The first literacy competition between artificial intelligence and humans was held at Novosibirsk State University. The result was better than the developers expected. The open system “Scribe”, developed by Ivan Bondarenko, a researcher at the Laboratory of Applied Digital Technologies of the International Scientific and Educational Mathematics Center of NSU, received a grade between three and four from the teacher.

Photo: freepik.com

As reported at NSU, the developers of «Scribe» it was important to collect statistics on the variety of errors he made in order to further improve the system. As a result, «Scribe» placed commas quite satisfactorily and divided the text into paragraphs, but hallucinations (a programmer term denoting the incorrect semantic meaning of a word) could not be avoided.

The work of the AI ​​was checked by Lyudmila Budneva – Senior Lecturer at the Department of Source Studies of Literature and Ancient Languages, NSU Humanitarian Institute.

“From 276 words of the dictation “Scribe” missed 6, five of which were at the end of the sentence, and in these cases he did not put a period, but began the next sentence with a capital letter, – Budneva reported. – In one place I missed the preposition “in”, which came second to last in the sentence. They heard 7 more words incorrectly. For example, instead of “highest” artificial intelligence wrote “revealed.” Another example of word creation — “calion-like” instead of «oilcloth». There was also the misheard expression “Read” I don’t want to.” Instead, it says “Count it or not,” which also indicates problems with grammar. There were also problems with grammar in writing endings — «blue» (instead of “blue” ones) and “portrait of… a schoolgirl” (correctly: “portrait of a high school student”), which is already counted as a spelling error….

As a result, experts made the following conclusion: in those places where the “Scribe” He heard all the words correctly, he wrote the dictation well — on the border between three and four. Its developers did not expect such a result.