To overwrite the CSS used by primefaces 5.x with your own CSS styles you need to define a Facet named "last" this will be interpretated by primefaces as last. Defining the new sytlesheet resource as child of this facet will result in a stylesheet loaded after all default stylesheet files allowing to override given styles with your custom styles.
<h:outputStylesheet library="css" name="custom.css" rendered="true">
The confusing part is the reference to the stylesheet. It´s expected in
Then the css file should be found by primefaces…