Elsa Margit Viola was born in Växjö 30 October 1906 to painter Johan Konrad Verner Bergman and his wife Selma Josefina Elisabet Danielsdotter. The family moved to Stenbrohult in 1910. In 1912 she had a little brother. In 1916 they moved to Snöstorp and in 1918 from there to nearby town, Halmstad.
Elsa Bergman had at least three children with bandmaster Lucillo Rizzi, who had a wife in Italy that he could not divorce, since he was a catholic. She was registered with her parents in Halmstad when the first two children were born and then moved to Varberg in 1933. Their daughter Renée was born in January 1928 in Falun - she was presumably following Rizzi on a tour. Her second child was born in Varberg, but it is uncertain if she ever resided there, because she is in Göteborg with the children at least from May 1933. From December 1933 to August 1934 Elsa and Lucillo lived in Nödinge, near Göteborg. From there they moved to Mariestad. In 1939 they arrived in Halmstad from Vänersborg.
The census 1940 finds them together in Halmstad.
Elsa Bergman is still registered in Halmstad in 1975. She is not present in 1985.