you are using wrong interface to do the job.
read the error very carefully. it says to put the pre-encoded compliant DVD-Video files in VIDEO_TS folder which are .VOB or .IFO.
while you are throwing .DAT, .MPG or .AVI in it.
it seems the interface you are using is just to make DVD Video Disk from a complete DVD video title but not from the files like you are adding.
if you see it is also recomending you to use Nero Vision to do what you are trying to do.
which version of Nero are you using?
Nero Vision can also be added in normal version of Nero as a plug-in any way.