Speer certainly had plenty of time to reflect and consider what he had to say in this book. It's a well written (though there seem to be more than a few typographical, grammatical or translational gremlins in the edition I have) account of his life from 1905 to 1946, cutting off as the verdicts of the Nuermberg trials were announced.
The impression I have is that Speer was an ambitious young architect in a time when there was precious little work around, and Hitler was an ambitious politician with vast architectural plans for Berlin and other German and Austrian cities. Speer claims political naïvety, which with hindsight is something of an understatement, but what other explanation could he give? He focused on architectural designs and ignored the "Jewish Problem", perhaps assuming Hitler couldn't possibly be serious, or at least that he couldn't do any real harm.
It is interesting to read how he gauged the changing popular mood in Germany in 1939, and how shocked they were that war was declared following the invasion of Poland. Nevertheless, Speer felt compelled to play a part in the war effort, despite Hitler's insistence that his building plans should continue unconstrained by the change in circumstance. In 1942 the Minister for Armaments was killed in a plane crash and Speer was appointed his successor, though he had been working on military architectural and engineering matters for some time before. In Speer's own mind this does not seem to have been a terrible turn of fate, but a natural choice following the accident. As a project manager and an engineer, Speer greatly improved the production of munitions, but in 1943 and 1944 became disillusioned with Hitler as a leader, perhaps as much because of the way he handled the bickering and intrigues within the Nazi leadership as the state of the war itself. Speer fell out of favour temporarily as he was named as a potential minister in paperwork related to the attempted assassination of Hitler in July 1944; further, he set about opposing Hitler's "scorched earth" orders as the German retreat gathered pace. He claims there were fanciful plans of his own to assassinate Hitler in 1945, but these never came to anything.
Speer insists on his ignorance of the Final Solution, noting only that a friend warned him "never to visit a concentration camp in Upper Silesia". Neither Speer nor his predecessor were present at the Wannsee Conference of January 1942. Speer's conviction at Nuremberg was on the basis of armaments production and the use of slave labour. kharin has bought me Gitta Sereny's book on Speer; it will make interesting future reading.