Couple weeks ago I got inspired to make a video using a free AI text to video generator. I selected Pika Labs, which generated 3-second videos.
My first attempt was Action Is an Option, and I simply used each line in the lyrics as a prompt. The results were very abstract and not that relatatable to the song, but interesting.
The second attempt was for Asteroid Blues, a collaboration of @ingolee Ingo and myself. This time I used Google’s Bard, a text generator AI, to read the lyrics of the song and generate the prompts I would use in Pika Labs, and without too much tweaking, got a video that is a bit more of a story but still plenty strange!
Then I had a brilliant idea to ask Bard to look in a folder in my Google Drive which contained 18 lyrics and mp3 files and create 18 scripts of prompts for all the songs. I also asked for help creating a Python script so I could type all of them into Discord to create approximately 1000 video clips in one fell swoop. Alas, Bard responded with total confidence it could do it. I repeatedly asked its progress, to which it responded it was ablout half through, let it go all weekend, and Monday it confessed it could not do it at all! So the idea will have to be shelved for now.
So today I had the idea to try a text to image AI generator and make a slideshow video, which I did using both Bing Chat and Canva. I used the verses as the prompts, and this became Love in a Warzone.
I made the videos with Cyberlink Powerdirector. And apparently its newest version has a built-in AI for who knows what!
Anyway, I kind of like Love in a Warzone the best so far, but the possibilities are more numerous everyday…
It is a big experiment for me, Jonathan. I did one more on my Youtube Channle, Expect It IS Your Right. What I am finding is video generation is very limited right now, so far very limiting on the number you can generate in a month, even for $10/month membership. I think the methods of generating 50-100 3 to 5 second clips ideally would be, hey AI, read my song lyrics, check the length of the mp3s, and make videos for every song in a folder. And then you decide what you don’t like, or maybe just have it generate 3 versions of everything and then use what you like best.
I am still going to try everything I can out there, but I could almost see Youtube automatically making music videos for just music in ayear or less.
Since I closed down the studio, I’ve been working full time with AI and machine learning for several years now. It’s my full time job at this point. I’ve actually designed and shipped some AI models that were based on youtube algorithms and designs. I assure you, google has the technology to do this right now. But I’m not clear on why they don’t use it.
Fascinating stuff this. But a lot of people are freaking out about where AI is heading @Jonathan , can you talk about what we might expect to see happening? What threats do you see? Super-smart viruses, cyberattacks for instance?
I suspect the answer is they are still pondering how to best monetize it while they are beta testing using the public and free use as enticements. I read an article that Microsoft recently has been making making rules for Dall-E so it will be unable to render politically/socially controversial prompts, like Mickey Mouse carrying an AR-15. Apparently the results so far are overly limiting the creativity for users. I can understand the need to find the happy mediums. We do not want next year’s elections overly fraught with fake news clips and no great way to manage it, for example.
I’m a big believer in free speech, but I also do not appreciate how the internet alone, much less AI, has been weaponized for diseminating misinformation and disinformation. It seems like a hopeless task I think to most people- the new normality. But with this wave of technology of AI, I think the companies realize they can and ought to try before as @ingolee pointed out the potential threats, even SkyNet scenerios, become our new new normality.
@ingolee - Sorry it took me a while to get back to you on this.
So AI seems to go through a lot of phases. For example when I first got started, NLP or ‘natural language processing’ was a hot new thing, now it’s old news.
At the moment everyone’s talking about generative AI. I just 2 days ago, got back from an ML (machine learning) conference in Boston where I defended some research I presented last year. There were a lot of companies there on panels predicting what this will mean for the employment of software developers. The overall takeaway was that it’ll shift trends in the developer market, but it won’t actually cut jobs.
Think of AI’s warring it out like battle-bots to brute-force crack each other’s security layers, or to out test and QA each other! lol. It’s quite fascinating if you think about computers that can write their own code, then battle it out against other computers on who can write the ‘smarter’ code!
@Jonathan I like that, I guess that’s reassuring; kind of like the ‘mutually assured destruction’ scenario of the cold war eh? We’re safe because nobody will dare unleash utter devastation to our power grid because our bots will destroy theirs? Thanks for the info.; it’s cool that you’re that doing research and are deeply involved in all this.
What do you say about the human side; I hear composers saying ‘ah but AI will never have the soul necessary to create truly great art’. Any thoughts?
I am excited about AI helping me fix my personal website in the near future. I did the one I have in VB code-behind style with SQL database and lots of SQL queries to pull up pictures, captions etc. But it broke a while back with a Godaddy update, and I’ve forgotten most of what I did originally and only have the general idea left. No matter, because I surely would modernize the way I handled it anyway. I am hoping that AI can write the code to do what I ask it to do, write the snippets and more. I will help and direct it as knowing what you want something to do determines what you get. For AI, knowing what to ask to elicit the best results is a skill and probably lots of trial and error.
I added another couple AI videos, one is Valentines Day BBQ. This one I used Invideo, again free. Its approach was different- it uses stock video footage to create an entire beginning to end video to accompany the song. I simply gave it the lyrics and told it the song length, and in a couple minutes it generated the entire video. Unexpectedly, however, it added a narration track in a Richard Burton-like voice that spoke about the lyrics very colorfully. You would likely think it was a human but for the fact it said BBQ literally many sentences instead of barbecue, probably 80 times! A human would never do that. I’m sure I could ask it to update it with “barbecue” and in a couple minutes it would be corrected, but that’s kind of the level of AI we have now- amazing but sometimes not quite getting on the first try.
Personally, I liked it saying BBQ, as if in England that’s what people normally say.
I am reading about WaveTool, an AI DAW, that assists in writing midi tracks and who knows what else. It wouldn’t surprise me if Reaper and Pro Tools and others get AI plug-ins either built in or not soon. Is it not valid if you use text-to-music generators to produce your tracks? What I think you wonder, Ingo, is if you merely ask the AI to write a great song, what would it come up with? If you said write a great Led Zepplin song. Grateful Dead. Beethoven.
I think we are soon going to know the answer to that. Right now I think it is hit and miss, and I don’t think AI can always discern the difference. But it does try very hard!
There are websites that will generate instrumental music to spec. although realistic vocals are still an issue as far as I know. Researchers have had AI write Beethoven style symphonies that sound good to many but some of the purists remain unconvinced due to the lack of ‘soul’ that they insist exists in true Beethoven works. It’s a subjective argument and I asked @Jonathan if he had any thoughts on that.
there is quite a push back against AI generated music and art when it comes to movie/gaming industry. For example I am not contractually allowed to use any AI created content for the productions that I am working on. Several distribution platforms (like Steam) will outright ban your content if AI created art is used.
Needless to say that there is a fight stemming against AI content due to blurred copyright lines. We will have to wait for judges for this one for a while for it to become a standard.
Traditionally artists avoided these issues by making enough small changes to create reasonable doubt about their theft since all artists are thieves of course . Except in rare instances if you create something totally original everybody will hate it so you have to make a connection to familiar material and then blur the lines. I’m sure AI can include an algorithm to do this extra step but like everybody else the developers have to rush to market with a beta version so as you say we’ll see how it plays out.
AI made content currently works well for home based studios and the hobby level productions. Bigger professional productions avoid AI content like plague currently. Though some may be secretly using it (As always), but they take a big risk while doing so.
It was quite the surprise for me when I had to legally agree to not using any AI art or content in production. Only then did I realize that this rabbit hole is only just opening, and it goes deep.
Since AI generated material cannot be copyrighted (supposedly) that certainly makes sense.
But how would anyone know without an audit of the production process? A passing resemblance to other existing works is quite common in all art. It takes a clear resemblance between two works to generate a lawsuit and even then they may not win in court.
If AI processes make straight digital copies from published material that seems like obvious theft. But if the AI goes deeper and looks for more general patterns and textures that is basically the same thing that a human artist does, perhaps subconsciously. But since the AI process is making use of copyrighted material to educate itself that is apparently not considered fair use, even though humans are doing that too.
In Indie Movies/ Video gaming industry, to be eligible for distribution on certain platforms (Netflix, Steam, Xbox) you have to agree to not use AI generated content. Then it is up to the platforms’ auditors to screen and decide…however anyone with a vendetta against the artist or producer can make a complaint and trigger a review. The work flow is such that you are taken down before you even get a chance to contest, and lasting damage is already done. You are at the mercy of the content distribution platforms.
it really depends on how big the artist is for Copyright to be major issue. The inconsistency of platform policies is the first major hurdle.
I suppose as artists we should be grateful that they are making an effort to prevent our work from being stolen. But I think this Genie is already out of the bottle.
As today marks the end of the writer’s strike, this is a good thing for real people vs AI as it deals with fair compensation for the use of AI images of actors. Nothing is perfect, but the issue is not to ban AI but regulate it. In music, Michelle, it must be much the same.
I am a big fan of virtual drums and guitar VSTs like Jamstix and UJAM. They are basically fancy loops of samples of real instruments that have powerful low level AI interfaces. I don’t have to write as many midi loops to create strumming and picking or drumming patterns manually, rather can ‘play’ them and focus on modifying parts. These programs/VSTs all point to use in film and music. The ability to compose melodies and arrangements is another set of VSTs I don’t use, but if I did I could be churning out songs I would feel not truly mine, therefore the borrowed/stolen AI inspirations of others, not me.
I understand how many real musicians I do not employ using midi, but I think of midi as an instrument I play. Since I could never afford to use real musicians for my songs, I feel I and everyone else using midi are not invalidated musically. Rather, it is the content we create that must be judged on its musicality.
I watched a documentary on Who’s Next and saw one part about Pete Townsend playing on an early synth that track on Won’t Get Fooled Again. He made a point he was manually twiddling the knobs to get the exact recording, which was justifying in his mind it was him not the machine making up that track, and I agree with him. However, now, I could create a track like that much more automatically, and I’m so much less talented than him.
I’m 71, and I can imagine having a personal care robot if I make it to 80, just to help me get in and out of bed and wash the dishes and feed the dogs. But I can imagine it also being able to play a guitar or drums or keys, even singing in any voice. I can see having a band comprised of robot AIs that could even perform live. I’d have to have a band name like Steve and the Robot Underlords. But I think, if I’m in charge, I would write the songs, at least some of them!
Hey man. Um… My take on that is it begs the question of what it is that people ultimately consume.
I’d say the ‘value’ is less about the creator, and all beholden to the eye of the consumer. What - other than the consumer, ultimately deems a ‘work of art’ to be worthy of their time, attention, and potentially cash? (Well… nah… forget about cash). Lets just say you hear a song and you like it. If you like it, at the end of the day, does it really matter if it came from a human or a bot?
Now ‘true greatness’… that’s a different question. Deep blue was great because it beat Kasparov in chess. Mangus Carlen also lost a chess game playing against his own app on that hardest difficulty level. So in the case of Deep Blue, was greatness not measured by the way it EXCEEDED human creativity?? (I know… hard pill to swallow haha).
@FluteCafe - you’re the tech guru out of all of us - I’m curious to hear your thoughts on this (feel free to dig out an old thread if you’ve already answered elsewhere).
Ha, I am hardly a tech guru but thanks for the kind words.
I do believe in the end result and quality of content being produced.
No human now can beat a modern day computer in chess (has not been done since 2005 Fritz vs Ponomariov)…and not just one human, computers today can beat 100 grandmasters playing together. So In a way the “end result” from a human perspective is better, because better computers will provide better training, and in the end a better player. So “training” was the end product there. Creative? I am not too sure about that…chess programs are hardly creative lol
In case of art, the end product is the art. But the issue with AI made art/music/games is the content that is used to train the AI is not tracked or regulated as of yet. People involved in training the AI, people involved in creating the training material (that may or may not be copyrighted) aren’t compensated properly in distribution. Leaving room for errors and possible lawsuits later. If the AI logic makes an error and leaves a “recognizable” footprint, all hell could break lose. This is why major platforms have not yet adopted AI. I would say it is a decade or two away to say the least.
. Except in rare instances if you create something totally original everybody will hate it so you have to make a connection to familiar material and then blur the lines. I’m sure AI can include an algorithm to do this extra step but like everybody else the developers have to rush to market with a beta version so as you say we’ll see how it plays out.
but thanks for the kind words.