Along with Taylor, Jeet Raval was batting on 29 off a mammoth 101 balls as New Zealand ended the first day on 85 for 2, trailing by 49 runs.
Windies finished the opening day on a high, having taken the wicket of Williamson cheaply.
Trent Boult took the remaining two wickets with Mitchell Santner effecting the run-out of Shane Dowrich.
Tom Latham was prolific at the top with a score of 37 off 87 balls, including five boundaries.
Sunil Ambrise was handed his Windies cap by Joel Garner.
Ross Taylor helped the home team stay calm and remained unbeaten on 12 at stumps.
Neil Wagner was brilliant with his career-best seven wickets as Windies were bowled out for 134.
Miguel Cummins took good advantage of the Wellington pitch with bowling figures of 6 runs in eight overs, including five maidens.
Kieran Powell was the visitor's top scorer with 42 off 79 balls, including eight boundaries.
Kane Williamson's run of poor form continued on from the India series with the New Zealand captain edging one off Kemar Roach to Shai Hope.
It was a terror debut Test innings for Ambrise who got himself dismissed hit wicket off Wagner for a duck.