At the time of enlisting he had served three years and was still serving with serving the 17 Ruahine Regiment. Embarked on 9 October 1915, with 1 Battalion, New Zealand Rifle Brigade, and disembarked at Suez on 15 November 1915. Re-embarked on the SS Ocean Princess on 19 December 1915 for Mersa Matruh. Returning from Mersa Matruh he disembarked at Alexandra, Egypt on 18 February 1916, before embarking on SS Arcadian for France on 6 April 1916.

Joseph Herzog was promoted to Corporal (in the field) on 10 May 1916 and attended Divisional Grenade School for a week commencing 23 July 1916. He returned to NZ Infantry base at Etaples on 15 August 1916 and from there he proceeded to England where he spent a few days from 31 August 1916 at the NZ Convalescent Camp, Hornchurch.

Promoted to Sergeant on 29 March 1917 and transferred to 4 Infantry Brigade the following day. On 15 April 1917 he was appointed Battalion Bombing Sergeant (3 Battalion) and left England for France on 28 May 1917.

In early 1918 he was appointed temporary Warrent Officer Class 2 and to temporary Company Sergeant Major (CSM S J Phillips wounded). On 21 February 1918 he was sent to England for officer training, starting a course of instruction at No14 Officer Cadet Battalion at Hipswell Camp, Catterick on 8 March 1918. On completion of the course he was promoted to Second Lieutenant on 10 September 1918.

He left for France on 25 September 1918 and joined 1 Battalion, Auckland Regiment on 5 October 1918. He was wounded in action on on 4 November 1918 and was admitted to 3 New Zealand Field Ambulance on the same day. He died of his wounds on 17 December 1918 at 20 General Hospital, Dannes Camiers, France.

His brother Corporal Charles Matthias Herzog (25/188) was killed in action in France on 25 May 1916 and is buried in Cite Bonjean Military Cemetery, Armentieres.